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ABSTBACT 



This comparison of the cost effectiveness of 
conventional, individualized, computer-assisted (CAI) and 
computer-managed instruction (ivil) for military training is based on 
data drawn from experiments of limited duration with relatively few 
students. All findings are confounded by effects that may be due to 
either CAI or CHI in comparison to conventional instruction, or to 
the revisions in course materials needed to modify a course from 
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conventional instruction, student attrition appears to increase with 
CAI or CWI as compared to conventional instruction: charges in 
student quality may account for this increase. H- w^s concluded that 
students prefer CAi or CMI to conventional instruction, attitudes of 
instructors are unfavorable to CAI and CMI, individualized 
instruction (without computer support) saves student time, and little 
additional student time is saved when the same courses are g^ven by 
CAI or CMI. cost savings attributed to CAI and CMI are based on 
estimates of pay and allowances of students for time saved by these 
methods. fAuthor/MEB) 
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ABSTRACT 



The cost and ef fectiveir^ss of computer-based instruction 
for military training are evaluated on the basis of about 30 
studies conducted since 1968. Four methods of instruction are 
distinguished and compared: 

Conventional Instruction : group-paced lectures, and 
discussions . 

Individualized Instruction ; self-paced (without computer 
support ) . 

Computer-Assisted Instruction (CAI) : computer stores and 
provides instructional materials to students individually 
via interactive terminals; computer tests and guides 
students; self-paced. 

Computer-Managed Instruction (CMI) : instructional materials 
and tests provided away from computer; computer scores 
the tests and guides students; self-paced. 

Much of the data come from experiments of limited duration and 
with relatively few students; by contrast, some CMI systems have 
been used for ^ years. All findings are confounded by effects 
that may be due either to CAI or CMI, in comparison to conven- 
tional instruction, or to the revisions in course materials 
needed to modify a course from conventional to CAI or CMI instruc 
tion. 

CAI and CMI save about 30 percent (median) of the time re- 
quired by students to complete the same courses given by conven- 
tional instruction; CAI and CMI cannot be compared directly 
because different courses were used in each study. Student at- 
trition appears to increase with CAI and CMI compared with 



ill 



conventional instruction, but changes in student quality may also 
account for this increase. Students prefer CAI or CMI to con- 
ventional instruction; attitudes of instructors, considered in 
only a few studies, are unfavorable to CAI and CMI. Individual- 
ized instruction (without computer support) also saveii student 
time; little additional student time is saved when the same 
courses are given by CAI or CMI. 

Direct comparisons of the cost and effectiveness of differ- 
ent methods of instruction are not now possible l^ecause only 
incomplete cost data were found. So-called cost savings attrib- 
uted to CAI and CMI are based on estimates of pay and allowances 
of students for the time saved by th<;se methods of instruction; 
allowances are seldom made for the costs of the CAI or CMI equip- 
ment and courseware, instructors, and other costs incremental to 
computer-based instruction. 
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SUMMARY 

This paper evaluates the cost and effectiveness of computer- 
assisted and computer-managed instruction for use in military 
training. The military Services have supported research and 
development on these methods of instruction since about I960. 

A. METHODS OF INSTRUCTION 

Methods of instruction can be placed conveniently in four 
groups, as described below: 

• Conventional Instruction , where an instructor may use 
lectures, discussions, laboratory demonstrations, and 
tutorial sessions. Groups of students proceed through 
the curriculum at the same pace; differences in achieve- 
ment among students are reflected in grades at the end 
of the course. 

• Individualized Instruction , where each student proceeds 
at his own pace through the curriculum that is arranged 
in a series of lessons and tests. Mastery of each lesson 
is set as a condition of progress. Differences among 
students are reflected in the amounts of time needed to 
complete the course, although grades may also be given. 
In general, an effort is made to assure about the same 
level of achievement for all students. 

• Computer-asslsted Instruction (CAI) . where all instruc- 
tional materials, i.e.^ lessons and tests, are stored in 
the computer; the student interacts with this material 
in real time via a terminal and display system. The 

1 
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computer can perform many functions ^ such as diagnose ^ 
student performance, prescribe lessons , maintain- records 
on student progress, and predict Individual course comple- 
tion dates. Current CAI systems differ In the number 
of terminals linked to a central computer (1 to 1000) 
and location of the central computer (which may require 
long-distance communications). In "stand-alone" sys- 
tems, a terminal and Its computer comprise the entire 
system. The PLATO IV system Is used In courses for 
medical technicians at Sheppard APB and for vehicle 
repair at Chanute APB. TICCIT Is used In courses for 
tactical coordinators for S-3A aircraft at Naval Air 
Station, North Island. GETS, a stand-alone system, will 
be used to handle training overloads in the TRIDENT 
program. 

Computer- managed instruction (CMI) . where instruction 
using self-paced lessons takes place away from the 
computer. The computer scores the tests and Interprets 
results to each student; advises him to take following or 
alternative lessons; recommends remediation; and manages 
student records, instructional resources, and administra- 
tive data. The Air Force Advanced Instructional System 
(AIS) is a prototype CMI system used for technical train- 
ing at Lowry APB. It can support up to 3000 students a 
day in four courses; the present version consists of 50 
student terminals (for scoring tests), 11 management 
terminals (for use by Instructors), and a CDC CYBER 73-16 
computer. The Navy Computer Managed Instruction System 
(Navy CMI) at Naval Air Technical Training Center, 
Mllllngton, Tennessee, now handles about 6000 students a 
day in 11 schools at five training centers in the United 
States; by 198O, it is expected to handle 16,000 students 
in 2k schools at 6 centers. 

2 
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B. APPLICATIONS OF CAI AND CMI IN MILITARY TRAINING 

,-CAI and CMI seem well-suited to providing specialized skill 
training both at military schools and at operational units in 
the field. Skill training at military schools is estimated to 
cost $3 billion a year and produce 1.1-million course graduates 
a year (FY 1979 data). The amount of technical training that 
occurs in operational units, i.e., away from formal schools, is 
thought to be large, but its magnitude is unknown; this includes 
on-the-job training, crew and unit training, refresher and up- 
grade training. 

The Department of Defense is estimated to spend about $12 
million a year for research and development on the use of com- / 
puters in military education and training (FY 1977 data). 

C. NATURE OF THE DATA ON COST AND EFFECTIVENESS OF CAI AND CMI 

The use of computer-assisted and computer-managed instruc- 
tion in military training has been evaluated in about 30 studies 
(producing i|8 sets of data) since 1968. Most (70 percent) of 
the data on CAI come from experiments with few students (up to 
50) and limited course materials (1 day to 1 week). There are 
fewer studies of CMI but these involve more students (600 to 2500) 
and longer courses (2 to 10 months). There is a wide range of 
subject matter in these studies, e.g., knowledge, theory, and 
hands-on performance skills; electronics machinist, recipe con- 
version, vehicle repair, fire-control technician. 

Each of the 30 studies report effectiveness. However, only 
eight of the studies which report effectiveness also provide 
some cost data. The latter data are limited to expenses incurred 
during the experiment and are incomplete with respect to costs of 
program management, maintenance and repair, instructional support, 
and other factors important in determining life-cycle costs. It 
is probably inappropriate to extrapolate from cost data in experi- 
ments to the costs of large-scale, long-term operational training 
programs . 

3 
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The effectiveness of training should be measured by how 
well course graduates perform specific Jobs In operational units. 
Instead, all studies use student achievement at school as a 
measure of effectiveness. The relation between achievement at 
school and performance on the Job Is essentially unknown, even 
for conventional Instruction. Data on length of time required 
for students to complete a course (generally less for CAI and CMI 
than for conventional Instruction) should be treated as a measure 
of the cost of Instruction rather than a measure of Its effective 
ness. The same argument applies to academic attrition rate. 
The attitudes of students and Instructors to CAI and CMI may be 
Interesting; however, they are qualitative In nature and It Is 
difficult to relate such data either to the cost or the effective 
ness of Instruction. 

The comparisons of alternative methods of Instruction are 
limited. Generally, CAI or CMI Is compared to conventional 
Instruction; we found only a few comparisons of CAI and CMI with 
Individualized Instruction (without computer support), a compari- 
son which relates to the benefits of computer support. In addi- 
tion, time savings found when CAI or CMI are compared to con- 
ventional instruction may be due to a combination of self-pacing, 
computer support, revised and possibly reduced amounts of course 
materials . 

D. EFFECTIVENESS OF CAI AND CMI 

Based on evidence provided by military research studies 
and qualified as noted above, the effectiveness of CAI and CMI 
Is evaluated as follows: 

• Student achievement . Student achievement at school with 
CAI Is about the same as that with conventional Instruc- 
tion In most comparisons and superior In about one-third 
of the comparisons. The differences In achievement are 
not thought to have practical significance. Student 
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achievement yrlth CMI is about the same as that with con- 
ventional Instruction. These findings are Important but 
also Inevitable because students are held In CAI and CMI 
courses until they achieve at least the standards estab- 
lished previously for conventional Instruction. 

• Student time savings . Students Instructed by CAI or CMI 
save about 30 percent (median value) of the time required 
to complete the same courses given by conventional In- 
struction. There Is a wide range In amounts of time 
reported as saved In these studies. The amounts of time 
saved by CAI and CMI cannot be compared directly because 
different courses were used for tests of these methods 

of Instruction. Where courses have been given for 
relatively long times, the Initial student time savings 
are maintained and, despite monthly fluctuations, tend 
to Increase. This finding Is based on four courses given 
by the Air Force Advanced Instructional System for about 
^ years and on three courses given by the Navy Computer 
Managed Instruction System for about 15 months; both 
systems are CMI systems. 

• Student attrition . The academic elimination rates In 
four courses on the Air Force Advanced Instructional 
System (AIS) appear to have Increased slightly over 4 
years compared to the previous base rates; however, the 
average academic elimination rate for all courses at 
Lowry AFB, i.e. y those not on AIS, Increased at the 
same time. Thus, the Increase In attrition may be at- 
tributed to AIS {i.e. y CMI) Instruction or to a decrease 
In student quality or to some combination of these two 
factors. Similar Increases in attrition seem to have 
occurred In six courses on the Navy CMI system over a 
15-month period; attrition dropped In one course; data 
on non-CMI courses for the same time period were not 
provided. 

5' 




• ftttitudes of students and instructors . Students In 
experiments almost always prefer CAI or CMI to conven- 
tional instruction. The attitudes of instructors are 
reported only in a few studies but these are almost 
always unfavorable to CAI and CMI in comparison to con- 
ventional instruction. 

• Time savings found with individualized instruction and 
computer-based instruction . Some data were found where 
the same course was given by conventional instruction, 
individualized instruction (i.e., self-paced instruction 
without computer support) and either CAI or CMI. Indi- 
vidualized: instruction saves student time. However, 

the adriltion of computer support (either CAI or CMI) to 
individualized instruction does not increase the amount 
of student time saved very much beyond that achieved by 
individualized instruction alone (i.e., without computer 
support). Again, differences between time savings attrib- 
uted to CAI and CMI cannot be evaluated because different 
courses were used in each group of studies. These data 
do not necessarily imply that the addition of CAI or CMI 
to individualized instruction (i.e., transforming the 
method of instruction) is not cost-effective. That would 
depend on whether the incremental costs of computer sup- 
port are offset by benefits in other areas such as, e.g.^ 
a need for fewer instructors and support personnel and 
for less administrative support. 

E. COSTS OF INSTRUCTION IN MILITARY TRAINING 

The benefits of computer-based instruction have to be 
compared with the cost of providing this type of instruction, 
but only incomplete cost data were found. 

• Collection of cost data . The military Services maintain 
systems that report the costs of individual courses. 



These are useful for such purposes as setting reimburse- 
ment rates for training students from other Services or 
other governments. They are not useful for analyses of 
the costs of different methods of instruction for the 
following reasons: (1) they do not distinguish the 
costs of parts of a course, which would permit determin- 
ing the costs of different methods of instruction used 
within a course; (2) costs of training support and 
management, that may vary considerably between methods 
of instruction, are allocated to individual courses on 
essentially arbitrary bases, such as the student load 
of all courses. 

Type of data needed on cost of Instruction . Each method 
of instruction in military training requires the expendi^ 
ture of funds for most, but not necessarily all, of the 
following functions: 

PROGRAM DEVELOPMENT 

Program Design 

Instructional Materials 

Conventional Instruction 
Individualized Instruction 

Prograjnming 
First-Unit Production 

Computer-Based Instruction 

Prograjnming 
Coding 

PROGRAM DELIVERY 

Instruction 

Instructors 

Instructional Support Personnel 
Equipment and Services 

Laboratory (including simulators) 
Media Devices 
Computer Systems 
Communications 



Materials (Including Consumables) 

Facilities 
PROGRAM MANAGEMEfiT AND ADMINISTRATION 
STUDENT PERSONNEL 

Pay and Allowances 

Other (Permanent Change of Station, 
Temporary Duty) 

Limited cost data were found for some of these resources 
and these are presented in the report. Cost data were 
not found or were extremely limited for the following 
resources for all methods of Instruction: 

- Program Design 

- Instructional Material: conventional instruction 

- Instructional Support Personnel 

- Laboratory Equipment 

- Materials (including consumables) 

- Program Management and Administration 

- Student Personnel: Permanent Change of Station, 

Tempbrary Duty, etc. 

• Collection of More Complete Data . Detailed cost data, 
required for analytical purposes, may be collected in 
three possible ways: 

- Universal, more complete reporting for all courses 
and support functions 

- Sampling selected courses and support functions 

- Ad hoc 

The. costs and benefits of these ways of collecting the cost 
data needed to evaluate alternative methods of instruction should 
be examined. 

F. COST-EFFECTIVENESS OF COMPUTER-BASED INSTRUCTION 

There have been few attempts to assess the cost-effectiveness 
of computer-assisted or computer-managed instruction and all of 
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se are limited, as indicated above, particularly with respect 
the cost data that have been used. The following results 
ve been reported: * 

• CM. The PLATO IV system was Judged to be not cost- 
effective in two evaluations. Although substantial 
amounts of student time were saved (19 to 89 percent in 
eight courses), PLATO IV was Judged to be not as cost- 
effective as self-paced instruction (because of high 
communications and maintenance costs) in one case and 
not as cost-effective as programmed instruction (because 
of greater development and operating costs) in the 
second case. 

CMj[. It was estimated that the Navy CMI system avoided 
costs of $10 million in FY 1977 and that the Air Force 
AIS avoided costs of $3 million in FY 1978. Both of 
these estimates are derived by translating amounts of 
student time saved into dollars avoided for student pay 
and allowances because of the reduced training times. 
The costs of providing CMI instruction are not considered 
in these reports, in a recent test, the AIS was Judged 
to be cost-effective, compared to instructor-supported 
self-pacing in one course but not in three others because 
of costs attributed to the AIS computer; however, the 
computer costs were small in comparison to other school 
costs . 



CONCLUSIONS 



A. MEASURES OF EFFECTIVENESS 

The effectiveness of computer-assisted and computer-managed 
Instruction for military training has been measured only by 
student achievement at school and not by performance on the Job. 
Correlations between performance at school and on the Job have 
not been established for any method of Instruction. 

B. STUDENT ACHIEVEMENT AT SCHOOL 

Student achievement In courses at military training schools 
with computer-assisted Instruction Is the same as or greater than 
that with conventional Instruction; the amount of additional 
achievement Is small and has little practical Importance. Student 
achievement In courses with computer-managed Instruction Is about 
the same as that with conventional Instruction. Both of these 
results are due to keeping students In CAI and CMI courses until 
they achieve standards set previously for conventional Instruc- 
tion . 

C. STUDENT TIME SAVINGS 

Computer-assisted and computer-managed Instruction In mili- 
tary training save about 30 percent of the time (median value) 
needed by students to complete the same courses given by con- 
ventional instruction. The amounts of time reported as saved 
vary widely, but little attention has been given to the factors 
that could account for the wide variation. Most of the results 
on computer-assisted Instruction come from experiments of limited 
duration, with limited amounts of course materials, and with 
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relatively few students. Where computer-managed Instruction has 
been used for extended periods (up to 4 years), the initial time 
savings have been maintained or increased. 

D. INDIVIDUALIZED AND COMPUTER-BASED INSTRUCTION 

Individualized instruction (self-paced instruction without 
computer-support) saves student time; little or no additional 
student time is saved when the same courses are given by computer-* 
assisted or computer-managed instruction. 

E. STUDENT ATTRITION 

Computer-managed instruction may increase the rate of student 
attrition for academic reasons, compared to that with conventional 
instruction. The observed increases in attrition may also be due, 
at least in part, to decreases in student quality, but this 
relationship has not been carefully examined. Student attrition 
appears not to increase with computer-assisted instruction, but 
this finding is based on tests of limited duration. 

F. STUDENT AND INSTRUCTOR ATTITUDES 

Attitudes of students toward computer-assisted and computer- 
managed instruction appear to be favorable. Attitudes of in- 
structors are reported as unfavorable, but this finding is based 
on very limited data. Little attention has been given to the 
role of instructors in computer-based instruction and to how they 
should be prepared for this type of instruction. 

G. COST DATA 

Only limited and incomplete data are available on the costs 
of computer-assisted and computer-managed instruction in military 
training. Data that are collected routinely on the costs of 
operational training programs are too highly aggregated, partic- 
ularly with respect to training support functions, for use in 
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analytical comparisons of computer-based Instruction with con- 
ventional instruction. 

H. COST-EFFECTIVENESS 

Estimates based on the amounts of student time saved sug- 
gest that the Navy Computer Managed Instruction System avoided 
costs of about $10 million in FY 1977 and that the Air Force 
Advanced Instructional System avoided costs of about $3 million 
in FY 1978. These estimates are incomplete because they do not 
consider the other costs of providing computer-managed instruc- 
tion at these installations or compare these costs with the costs 
of alternative methods of instruction for the same courses. 
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RECOMMENDATIONS 



A. JOB-PERFORMANCE DATA 

Improve methods currently available for measuring perform- 
ance on the Job In areas related to technical training. Compare 
achievement at school with performance on the Job for students 
In courses given by computer-assisted and computer-managed In- 
struction; to whatever extent opportunities exist, do the same 
thing for the same courses given by conventional and Individual- 
ized Instruction. The Job-performance data should be collected 
for several time Intervale after students leave school to deter- 
mine whether benefits In favor of any method of Instruction are 
sustained as Job experience Increases. 

B. COST AND EFFECTIVENESS DATA 

Evaluate alternative methods of collecting reliable data on 
the costs and effectiveness of Instruction In military training. 
Based on these findings, develop and Initiate data-collection 
programs on the costs and effectiveness of alternative methods 
of Instruction. 

C. RESEARCH AND DEVELOPMENT PLAN 

Bring up to date the "Integrated Department of Defense Plan 
for Research and Development on Computers In Education and Train- 
ing" (Department of Defense, September 1975). Support Is needed 
for Exploratory and Advanced Development (6.2 and 6.3 RDT&E 
funds) on many subjects Identified In this paper, such as the 
development of objective measures of performance on the Job, 
comparisons of student achievement at school with performance on 
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the Job, the development of methods to measure the quality of 
course materials and delivery of Instruction, and studies to 
account for the relative contributions of self-pacing, course 
revision, computer support, and other factors to the amounts of 
student time saVed by computer-assisted and computer-managed 
Instruction. Support for other studies to Improve various 
aspects of computer-assisted and computer-managed Instruction 
may well be questioned' until more reliable cost data are avail- 
able to determine areas of high pay-off. 

D. CURRENT AND PROPOSED SYSTEMS 

Collect data on the costs of Instruction for courses and 
course segments given now by computer-assisted or computer- 
managed Instruction for military training, e.g., PLATO IV at 
Sheppard Air Force Base, Texas, and at Chanute Air Force Base, 
Illinois; TICCIT at North Island Naval Air Station, San Diego, 
California; Advanced Instructional System at Lowry Air Force 
Base, Denver, Colorado; and Navy Computer Managed Instruction 
System at Naval Air Technical Training Center, Mllllngton, 
Tennessee. Comparable baseline cost data should also be col- 
lected, as far as possible, for alternative methods of Instruc- 
tion for the same courses. Projections of cost should be made 
for computer-managed Instruction systems that are now being 
planned; i.e., the Navy Aviation Training Support System, the 
Army Automated Instructional Management System, and the Marine 
Corps Communication-Electronics School CAI/CMI System. 

E. RANGE OF TIME SAVINGS 

Determine the factors which account for the large variations 
In the amounts of student time saved by computer-assisted and 
computer-managed Instruction In various studies. Consideration 
should be given to such factors as quality of courseware (In- 
cluding that In conventional courses). Instructional strategy, 
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types of subject matter presented in courses, and the amount and 
type of guidance provided by Instructors. An effort should also 
be made to resolve the extent to which such factors as self- 
pacing, course revision, shortening courses, and various types 
of computer- support contribute to the total amounts of student 
time saved. 

F. STUDENT ATTRITION 

Determine the extent to which observed increases of student 
attrition with computer-managed instruction are due to this method 
of instruction and to other factors that may also be present, 
such as changes in the quality of students. 

G. INSTRUCTORS' ATTITUDES 

Determine the attitudes of instructors to computer-based 
and other methods of instruction in a systematic manner so that 
remedial actions can be taken as required. 
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I. INTRODUCTION 



A. PURPOSE 

The purpose of this study Is to evaluate research and devel- 
opment on the cost-effectiveness of computer-based Instruction 
for military training. 

The use of computers to provide and support Instruction Is 
the result of significant developments that have occurred since 
about 196O: (1) growth In the capabilities of computer hardware 
and software and (2) Improved procedures for designing lessons 
In a self-paced or Individualized format needed for computer- 
based Instruction. The Department of Defense and the military 
Services have supported the development of computer-based Instruc- 
tion because of Its potential value to Improve the effectiveness 
and reduce the cost of training, particularly where large numbers 
of students are Involved. 

This study was performed for the Deputy Under Secretary of 
Defense for Research and Engineering (Research and Advanced Tech- 
nology), Office of the Under Secretary of Defense for Research 
and Engineering. It responds to a recommendation made by the 
Defense Science Board:* 

To Improve the effectiveness of training and 
training technology R&D, the DoD should: 

1. Develop a capability to perform cost- 
effectiveness analyses of training 
technology. 



•Summary Report of the Task Force on Training Technology, 
Defense Science Board, 27 February 1976, (p. x). 
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B. METHODS OF INSTRUCTION 

The Rand Corporation's "Method of Designing Instructional 
Alternatives (MODIA)" Identifies 20 different methods of teach- 
ing (Carpenter-Huffman, 1977). For convenience, methods of In- 
struction are organized here In four groups; more than one method 
of Instruction may be used In a course. (See Appendix A for a 
more complete discussion.) 

1. Conventional Instruction 

Conventional Instruction refers to many possible combina- 
tions of lectures, discussions, laboratory, and tutorial sessions 
as a method of Instruction. A key feature of conventional In- 
struction Is that groups of students proceed through a course at 
the same pace. Differences In the amount of Information retained 
by students are reflected In their grades at the end of the 
course. Conventional Instruction Is used In 75 to 90 percent of 
all military courses, although a precise estimate is not avail- 
able. It Is also referred to as lock-step Instruction, platform 
Instruction, and group scheduling. 

2. Individualized Instruction 

In Individualized Instruction, a course Is arranged In a 
series of lessons and tests and each student proceeds at his own 
pace. Mastery of each lesson Is prescribed as a condition of 
progress. Differences among students are reflected In how long 
It takes them to complete a course, although grades may also be 
given. 

There are various forms of Individualized Instruction that 
differ primarily In such ways as the structure of lessons pro- 
vided to the student (main line, branching) and the extent to 
which the student Is completely free to proceed at his own pace. 
All methods of computer-based Instruction rely on some form of 
Individualized Instruction; by definition, the term "Individual- 
ized Instruction" will be used here to apply only to this method 
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of instruction conducted without computer support. The terms 
individualized instruction, self-pacing, and programmed instruc- 
tion will be used synonomously unless otherwise specified. 

3. Computer-Assisted Instruction (CAI) 

In this paper the term computev^baeed inetvuction refers 
generally to both CAI and CMI methods of instruction. In 
computer-assisted instruction (CAI), the student interacts in 
real time, via an interactive terminal, with instructional 
material that is stored in the computer. This offers great 
flexibility for presenting alternative versions of the same les- 
sons according to each student *s particular way of learning. 
Most CAI systems diagnose student performance, prescribe les- 
sons, and maintain' student records. Examples of some CAI systems 
follow (see Appendix B for a more complete discussion): 

• PLATO: Programmed Logic for Automatic Teaching Opera- 
tion. A current version of this system, PLATO IV, can 
support about 950 terminals linked through microwave and 
land-line communications to a large central computer 
(CDC CYBER 74) located at the University of Illinois. 

• TICCIT: Time-Shared Interactive Computer-Controlled 
Information-Television. The basic TICCIT system uses 
one or two mini-computers to support up to 128 terminals 
at one location. 

• LTS : Lincoln Terminal System. The latest version, LTS-5, 
uses microfiche to store both visual images and an audio 
track. This is a self-contained or "stand-alone" system. 

• GETS : General Electric Training System. This is a 
stand-alone system which uses a random access 35-mm slide 
projector for visual displays and floppy discs for les- 
son preparation and playback. 

4. Computer-Managed Instruction (CMI) 

In "computer-managed instruction (CMI), instruction takes 
place away from the computer. The computer scores tests and 
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Interprets results to the student; advises on following or alter- 
native lessons; recommends remediation; and manages student re- 
cords, resources, and administrative data. (See Appendix B for 
a more complete discussion.) 

This process Is Initiated typically when the student places 
a test answer sheet on an optical reader connected to the central 
cpmputer. He receives the results on a printout which tells him 
how well he performed, what lesson to take next, and where to 
find It. Examples of some CMI systems follow: 

• AIS: Advanced Instructional System. This prototype 
system Is Installed at the Air Force Technical Training 
Center, Lowry Air Force Base, Denver, Colorado. The 
present version consists of 50 student terminals, 11 
management terminals, and a CDC CYBER 73-16 computer 
which can support up to 3,000 students a day In four 
courses. These courses were selected to represent a 
cross section of the technical training courses at Lowry 
AFB and serve about 25 percent of the student body there. 
The management terminals provide CAI services for use by 
Instructors (for developing or revising lessons and for 
retrieving data collected by the system) . The system 
could be expanded to provide CAI services to students. 

• Navy CMI; Computer Managed Instruction System. This 
system. Installed at Naval Air Technical Training Center, 
Mllllngton, Tennessee (also referred to as Memphis, 
Tennessee), handles about 6,000 students In 11 schools 

at 5 centers. It Is based on a Honeywell Series 6o, 
level 66 computer. 

• CTS: Computerized Training System. This system can 
provide CAI and CMI services for 128 terminals at the 
U.S. Army Signal Center and School, Fort Gordon, Georgia. 
It Is based on six mini-computers (PDP-ll/35s ) . Each 
terminal contains a visual display unit and a keyboard 
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which can provide both Interactive Instruction and course 
management services. (Note: A report evaluating the CTS 
In a CMI mode arrived too late for use In this paper. 
See Seldel, Rosenblatt, Wagner, Schulz, and Hunter, 
1978.) 

C. DISTINCTIONS BETWEEN MILITARY AND NON-MILITARY INSTRUCTION 

Military personnel receive pay and allowances while they are 
In training. Thus, any procedure which can reduce the length 
of time required for training, without significantly affecting 
the amount and/or quality of Information acquired, can assist In 
reducing the cost of training at military schools; It can also 
result In Increasing the amount of time spent by military person- 
nel In operational assignments during their military careers. 
Military training courses are designed to qualify students for 
well-defined Jobs to which they can be assigned upon successful 
completion of these courses. 

The situation differs In almost all types of public and 
private education where students remain at school for required 
periods of time and are not paid while being Instructed. These 
schools receive no direct benefits for completing Instruction In 
less than the required time. Courses are generally not designed 
to qualify students for particular Jobs and, obviously, schools 
cannot assign students to Jobs when they graduate. 

A major consequence of these distinctions Is that methods 
of Instruction that are cost-effective for military training 
may not be cost-effective In other areas. Another Is that re- 
search on computer-based Instruction supported by the military 
Services has emphasized the possibility of saving student time 
while maintaining student achievement constant. Research on 
Instruction In non-mllltary settings has been concerned more 
with the amount of student achievement at the completion of a 
course than with the amount of time needed by students to acquire 
+'-he material. 
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D. APPLICATION OF CAL AND CMI 

The potential application of CAI and CMI would appear to be 
primarily for specialized ■ skill training at technical schools 
which prepare military personnel for specific Jobs In the mili- 
tary Services. About 300,000 people complete recruit training 
each year and become candidates for specialized skill training. 
Skill training Is estimated to cost $3 billion and to produce 
1.1 million course graduates each year (1211,000 man-years of 
training In FY 1979); about 79,000 people (75 percent military) 
are needed to conduct and support this training (Department of 
Defense, Military Manpower Training Report for FY 1979, March 
1978). About 75 percent of all training loads are for new 
accessions to the military Services. 

CAI and CMI also appear appropriate for certain types of 
training that occurs away from formal schools, such as on-the-job 
training, crew and unit training, refresher and upgrade training 
In .operational units. The magnitude of these efforts Is thought 
to be large but no estimate of Its cost has been made. The 
"Integrated DoD Plan for R&D on Computers In Education and 
Training", prepared by a trl-servlce group In September 1975, 
proposed that $12.1 million be allocated In FY 1977 for research 
and development on coi|iputer-based Instruction. An estimate of 
the funds allocated by the DoD to R&D on computer-based Instruc- 
tion In recent years has not been made. 
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II. PROBLEMS ENCOUNTERED IN EVALUATING THE COST EFFECTIVENESS 
OF COMPUTER-ASSISTED AND COMPUTER-MANAGED INSTRUCTION 

The military Services have provided strong support to re- 
search and development on computer-based Instruction since the 
early 1960s because of Its obvious application to military train- 
ing. This same time period saw the development of new tools for 
analysis and management of military (and other government) re- 
sources. The analytic procedures are best described as the 
adaptation of traditional economic analysis to government opera- 
tions. 

Traditional economic analysis Is Identified with production 
processes In which organized markets exist for determining the 
values of both resource Inputs and outputs In a common unit of 
measure, such as dollars. In military activities, resource In- 
puts are typically obtained from organized markets and valued In 
dollars, but no such market exists for determining the dollar 
value, of ^resource outputs (e.g., the cost of military training 
may be^ determined, but what dollar value should be placed on Its 
results?). The lack of comparability between inputs and outputs 
In economic analyses of military systems has led to the develop- 
ment of special analytic techniques. Cost-effectiveness analysis 
is one of these, and it has become a general requirement for the 
management of military resources . 

A. REQUIREMENT FOR COST-EFFECTIVENESS ASSESSMENT 

Department of Defense Instruction (DODI) 70111.3, Economic 
Analysis and Program Evaluation for Resource Management (1972) 
establishes the general policy for cost-effectiveness analyses 
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and outlines the requirements to which such analyr>es must adhere. 
Cost-effectiveness analyses are required for first-time funding 
of projects and periodically for on-going activities. This policy 
has been promulgated In each of the Services by Implementing 
Instructions and has been Interpreted In numerous writings. 

There are two ways of evaluating the cost-effectiveness of 
alternative military systems. Given two systems of the same 
cost, one would prefer the system that provides greater effective- 
ness. Given two systems of the same level of effectiveness, one 
would prefer the system that costs less. All studies of computer- 
assisted and computer-managed Instruction have used the second 
approach. Computer-based Instructional systems have been designed 
to provide the same degree of effectiveness (student achievement) 
as the method of Instruction they might replace (conventional 
Instruction). Therefore, these alternative methods of Instruc- 
tion must be evaluated In terms of differences In their costs. 

To date, evaluations of computer-assisted and computer- 
managed instruction have addressed questions concerning the tech- 
nical and operational feasibility of these methods of instruction, 
including the : jsign of courses for these methods of instruction. 
Most studies have addressed the effectiveness of instruction; 
some treated costs and some treated cost-effectiveness. However, 
cost and cost-effectiveness appear to have been secondary con- 
siderations in these studies. Table 1 lists 30 studies that gen- 
erated ^8 data sets on the effectiveness of CAI or CMI in military 
training; only eight of these provided any data on the costs of 
these programs, most of which were experimental rather than opera- 
tional in nature; only five evaluated cost-effectiveness. 

At some point, the cost-effectiveness of computer-based 
instruction will have to be established in a definitive fashion. 
Taken as a group, these 30 studies do not provide a sufficient 
basis on which to evaluate the cost-effectiveness of either 
computer-assisted or computer-managed instruction. 
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TABLE 1. DATA ON EFFECTIVENESS AND COSTS IN EVALUATIONS OF 
COMPUTER-ASSISTED AND COMPUTER-MANAGED INSTRUCTION 





SystM 


S#fVi09 


ItttWw 




Naaifear af Evalaatlaas 
(Data fats) 








EflacUvaaaat 


Cast 


Cast- 
EfNcttvaaasi 




Ml 1S00 


A 


Amy SIfHl Ctilir ft SckMl, 
Ft MmmmUi 


m(ms) 

Liii«t (1N9) 

BkMi ft Lfi|i (1971a) 

OMi ft Uiia (19711) 


I 








■Him 


N 


NNIOC 


Fart ft tiaefl (1970) 
Harfack ft M ;1971) 

Fort, ttomli, ft Harfack (1972) 


I 


1 






PLATO nf 


A 


Amy MftMCt CMftr A 
SdMsif AStevtffMi P6 


U.9. Affliy Oriaaaea Caatar ft ScM 

(1979) 




1 


1 


CAI 


PLATO IV 


N 


NPMG 
Sm Dtoft 


tiara (1975) 

«aafliftCaiy(aa9aMsM) 

Lafeay. Crawfsfi. ft HMtocfc (1979) 
Crawfartit Hnrtick, af aC (1976) 
FriMsto ft Hatvar-Mca (1977) 
HwtM* ft ttovtli (1976)* 


\ 


1 
1 


1 




PLAT^ IV 


AF 


SiMfM AFB 


ttaialtawlaaf, Daigaaa, araC (1977) 
Daifaaa ft Daacan (1977) 


j 


1 






PLATO IV 


AF 


CtUMrte AFB 


Dalaua.DaLH.«raC(1977) 




1 


1 




PLATO IV/ 
DMMI 


N 


fiiMtd Mull ScliMl, 
DMiNtck, VA 


eaaaral Bactrtc Ortfaaaca lystan (197S) 
MkiaaftfirMsaa (1979) 


I ; 








LTS-3 


AF 


KMSter AFB 


Harris, firaukarif. fffoL (1972) 
Dawas, Jatesaa. «r aC (1972) 
Kaatlar AFB (1972} 
Kaatir AFB (1973) 
Kaatlar AFB (1974) 




1 






IXCTT 


N 


NPflDC, San CXaga 


Walkar (1978) 








CM 


Navy CM 


N 


NATTC 
»■ — 
Mviiipnii 


Cartaa, firahaai, «f (1975) 




1 


1 




Att 


AF 


AFNflL*n 
Lnrry AFB 


Att BrMIng (1978) 


4 




1 



**0k^7avSMU iMftStM pflSfMMMtf lilt. 



ERIC 



27 



40 



There is an extensive literature that describes procedures 
for conducting cost, cost-effectiveness, and cost-benefit analy- 
ses, e.g., Sassone and Schaffer (1978), Quade and 3oucher (I968), 
and Fisher (1971). Applications of these procedures to a variety 
of fields may be found in Alf andary-Alexander (I968) and Goldman 
(1967). The application of these procedures to military train- 
ing is described by Doughty, Stern, and Thompson (1976) and 
Swope (1976).' Resource estimation procedures associated with 
the conduct of military training are identified, among others, 
by Hess and Kantar (1977) and Braby, Henry, Parrish, and Swope 
(1975). 

B. CRITIQUE OF THE LITEP^T'RE 

Computer-assisted ai. ^mputer-managed instruction in mili- 
tary training have been evaluated in about 30 studies conducted 
since I968. Most of these were experiments conducted with R&D 
funds, while a few approximated operational conditions. Most of 
these studies were concerned primarily with the effectiveness of 
computer-based instruction, a few with its costs; some implied 
that their results related to cost-effectiveness but did not 
actually perform any analyses. The critique that follows discus- 
ses the following issues: 

• The scope of the studies 

• The measures of effectiveness used 

• The incompleteness of cost information 

• Treatment of expenditures as costs 

• The incomplete range of alternatives considered. 
1. Scope of the Stuoies 

Most of the data were collected under programs funded through 
the RDT&E appropriation, i.e.. Exploratory Development (6.2) and 
Advanced Development (6.3). Such programs are generally small 
in scale with regard to numbers of students, hours of instruc- 
tion, and duration. The i\8 data sets developed in these programs 
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are based on totals of about 800 hours of computer-based instruc- 
tional materials, 9,000 students, and 400,000 student hours. 
The four data sets for the Advanced Instructional System (AIS), 
a demonstration rather than an experimental program, account for 
approximately llO percent of all instructional hours, over 70 per- 
cent of the students, and over 85 percent of the student hours in 
these 30 studies. Of the remaining kl\ programs, only 21 involve 
more than 10 hours of instruction and only 18 include more than 
50 students as subjects (see Table 2). 

TABLE 2. DISTRIBUTIONS OF COURSE LENGTH AND NUMBER OF 
STUDENTS IN 48 EVALUATIONS OF MILITARY CAI 
AND CMI INSTRUCTION 
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NOTE: Al entries In table refer to CAI except where two values are shown. Then, read *XAI/CMr. 
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The results obtained with respect to student achievement and 
time required to complete courses in short-term experiments may 
differ from those found in large-scale, long-term operational 
training programs. Some data in Chapter III suggest that similar 
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results are found In both cases. However, the extrapolation of 
cost data from experiments to operational programs Is partic- 
ularly Inappropriate. Management and accounting of resources 
differ between operational programs and those funded through 
Research, Development, Test and Evaluation. In addition, dif- 
ferences In cost may be anticipated between training In opera- 
tional and In research settings because of different scales of 
operation, different organization of the training program, and 
differences In the utilization of equipment and personnel. 

2. Measures of Effectiveness Used 

The purpose of military training is to provide personnel 
with the skills and knowledge required to perform specific tasks 
In the operational forces under both peacetime and combat situa- 
tions. Thus, the effectiveness of alternative methods of In- 
struction must be evaluated by comparing how well personnel, 
trained by either method, perform In operational units. Meas- 
urement of performance, of graduates on Jobs In the field Implies 
a system for postgraduate monitoring of students for some period 
of time after their assignment to duty stations. This measure 
\ias not used In any of the studies. At present, data on the 
effectiveness of training In schools are not collected system- 
atically; the data that are collected consist of supervisors' 
opinions about the .job performance of graduates: Such data are 
subjective In nature and may be Influenced by factors not related 
to training, e.g. relevance of the training course to the actual 
job, nature of the work environment, personality, and so on. 

Instead, we found that the following measures of effective- 
ness were used In these studies: (i) student achievement on 
tests administered during and/or at the end of course, (2) the 
length of time required for students to complete a course, 
(3) academic attrition rates, and (H) student and Instructor at- 
titudes. None are appropriate measures of effectiveness. The 
use of multiple measures may lead to contradictory conclusions, 
unless they can be combined In a meaningful way. 
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Training course curricula are based on assessment of skills 
required of personnel In operational billets as determined, for 
example, by a task analysis. To the extent that such assessments 
are valid, student achievement In school as shown by test results 
may serve as a proxy (or predictor) of future field performance 
and, hence, training effectiveness. However, correlations between 
performance at school and on the Job have not been established for 
any method of Instruction, and the use of results collected only 
at schools cannot be taken to be conclusive. On the whole, the 
data suggest that student achievement at school Is about the same 
with all methods of Instruction considered In this paper. The 
differences that were found are not thought to have practical 
Importance. This will be discussed later In this paper. 

To the extent that student time Is relevant to the analysis 
of cost-effectiveness. It Is a measure of the cost and not of the 
effectiveness of training. The studies would have gone a long 
way toward fixed-effectiveness or net cost evaluations If the 
observed decreases In the time needed by students to complete 
courses given by computer-based Instruction had been converted 
to decreased (or avoided) cost of Instruction, through standard 
factors for pay and allowances and other personnel-related re- 
sources; these decreases would have to be offset against the 
costs of other resources that are Incremental to the use of 
computer-based Instruction, e.g. y computers and courseware. Cost 
Is similarly associated with student attrition. With lower at- 
trition rates, fewer students are required to enter training to 
produce a specified number of graduates and, thus, a smaller 
total number of student days are spent In training. 

Student and Instructor attitudes are qualitative and not yet 
quantifiable factors. To the extent that student attitudes might 
Impact on school achievement. It might be manifest In test per- 
formance, course time, and attrition; these possibilities are 
not known and have not been explored. While It may be granted 
that attitudes might affect either cost or effectiveness through 
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such considerations as later field performance and reenllstment 
rates (Including those of Instructors), the relationships, If 
any, are oblique and remain unknown. 

3« Incompleteness of Cost Information 

In the eight studies that addressed cost, the cost data 
described only some of the direct expenditures that were Incur- 
red during the course of the studies. Some of this Information 
appears to have been reconstructed after the fact rather than 
recorded during the period of experimentation and It may not be 
accurate. 

Training Is an Intricate process that requires different 
types of resources to perform a variety of functions. Some re- 
sources may be uniquely associated with a single method of In- 
struction while others will be common to several methods. For 
example, computer hardware (a type of resource) Is a unique 
requirement of computer-based Instruction. Development of 
courseware (a function provided by resources) Is associated with 
all methods of Instruction, but Its cost per unit (e.g., man- 
hours per hour of Instruction developed) appears to vary widely 
between different methods of Instruction. Similarly, Instruc- 
tional personnel are employed by all methods of Instruction; 
while Its nominal cost (per hour of Instruction) may be constant 
between Instructional methods. Its effective cost (per student 
hour) depends on the student .-Instructor ratio characteristic of 
each method of Instruction. In comparing the costs of alterna- 
tive methods of instruction, it is necessary to account for all 
resources whose costs may differ between alternatives. That is, 
all such costs must be considered relevant to the analysis. 

It is a relatively straightforward exercise to Identify 
the resources for which data, are needed to compare the costs of 
computer-based Instruction and of other methods of Instruction. 
Table 3 displays a list of these resources, at a major category 
level, developed from our reading of the literature. None of 
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TABLE 3. RESOURCES REQUIRED TO SUPPORT 
VARIOUS METHODS OF INSTRUCTION 
(MAJOR CATEGORIES ONLY) 



RESOURCE (TYPE OH FUNCTION) 

PROGRAM DEVELOPMENT 
PROGRAM DESIGN 

INSTRUCTIONAL MATERIALS 
CONVENTIONAL 
INDIVIDUALIZED INSTRUCTION 
PROGRAMMING 
HRST-UNIT PRODUCTION 
tJMPUTERBASED 
PROGRAMMING 
CODING 

PROGRAM DEUVERY " ~ 

INSTRUCTION 
INSTRUCTORS 

INSTRUCTIONAL SUPPORT PERSONNEL 
EQUIPMENT AND SERVICES 

LABORATORY (INCLUDING SIMULATORS) 
MEDIA DEVICES 
COMPUTER SYSTEMS 
COMMUNICATIONS 
MATERIALS (INCLUDING CONSUMABLES) 

FACILITIES 

PROGRAM MANAGEMENT AND ADMINISTRATION 
STUDENT PERSONNEL 

PAY AND ALLOWANCES 
OTHER (TEMPORARY DUTY, 
PERMANENT CHANGE OF STATION. ETC.) 
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the eight studies which reported costs of computer-based Instruc- 
tion provided data for all Items on this list. 

The extent of Incompleteness may be Judged by the following 
comments. Computer hardware (either leased or procured) and 
courseware development are two categories of major Impact assoc- 
iated with the cost of computer-based Instruction; three studies* 
provided no Information regarding courseware costs and two^ pro- 
vided no Information regarding computer hardware costs. Only 
four of the studies provided cost Information for anything other 
than computer hardware or courseware development'; In two of these 
cases, the only other costs reported were for compressed air and 
carrels for PLATO IV terminals, relatively minor Items." 

More notable than the Incompleteness of data on the costs 
of computer-based Instruction Is the lack of Information regard- 
ing the costs of alternative methods of Instruction. All studies 
compared the effectiveness of computer-based and an alternative 
method of Instruction (generally conventional Instruction), using 
the measures described abx)ve. Only one of the eight studies ad- 
dressing costs (Crawford, Hurlock, et al., 1976) compared the cost 
of the experimental program to that of the method by which the 
same material was normally taught. Two studies provided In- 
complete Information on courseware development for Individualized 
Instruction (U.S. Army Ordnance Center and School, 1975; Dallman, 
DeLeo et al. , 1977); the data were taken from other studies. In 
essence, even when some data were provided on the costs of 



»Ford, Slough, and Hurlock (1972); Crawford, Hurlock, et at. 

(1976) ; Stelnkerchner, Delgnan, et at. (1977). 

^Carson, Graham, et at. (1975); Keesler AFB (197^). 

'Crawford, Hurlock, et al. (1976); Carson, Graham, et al. (1975); 
Stelnkerchner, Delgnan, et al. (1977); Dallman, DeLeo, et al. 

(1977) . 

"Stelnkerchner, Delgnan, et al. (1977); Dallman, DeLeo, et al. 
(1977). 
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computer-based instruction, comparable cost data were not pro- 
vided for alternative methods of instruction. 

4. Expenditures as Cost 

Most studies that considered the cost of training treated 
expenditures of funds during the course of the experimental pro- 
grams as equivalent to the cost of training. For example, in ex- 
periments using PLATO IV, no distinction was made between funds 
expended for the purchase of terminals and for access to the 
central processor. The expenditure for terminals is an invest- 
ment in long-lived assets that can provide training both during 
and after the period of the experiment; thus ^ only a fraction of 
the procurement cost is a cost of training, i.e., during the 
limited time of the experiment. On the other hand, expenditure 
for purchase of central processor time is strictly a cost of the 
experimental program; access to the central processor during the 
experiment provided no residual capability to support training 
after the experiment was completed. The simple sum of expendi- 
tures for terminals (investment) and accest^ to the computer 
(operations) is meaningless for any pe?.''iod less than an assumed 
total life-cycle of the system. Two stud; (Hurlock and Slough, 
1976 and Crawford, Hurlock, et at. ^ 1976) \^ere exceptions to 
this type of treatment. In these cases, a DOrtion of the re- 
corded expenditures was translated into e ...Imated costs of an 
operational PLATO IV training pi^ograr . 

Translations from current expenditures to costs that can 
be summed into meaningful totals require resort to some form of 
analytic framework or model, and a formal model is called for 
in cost-effectiveness analysis. It Imposes the discipline of 
explicitly identifying all inputs, assumptions, and relation- 
ships so that alternatives can be compared in a consistent manner. 
For example, alternative methods of instruction may be affected 
differently by such conditions as limits on utilization of equip- 
ments (e.g., attainable terminal hours), availability of required 
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resources, {e.g., media devices) and differences in lifespans 
of various resources {e.g., computer hardware versus laboratory 
equipment); yet, such conditions are difficult to treat completely 
and consistently. Models which would call attention to the full 
range of inputs and assumptions and the ways they are incorporated 
into evaluations of cost were missing from the studies reviewed. 
5- Incomplete Range of Alternatives Considered 

All studies compared student achievement and the times 
needed by students to complete the same course given by conven- 
tional instruction and by computer-based instruction. A course 
given by conventional instruction must be rearranged into a series 
of lessons and tests in order to be given by computer-assisted or 
computer-managed instruction, or by individualized instruction 
without computer support. During the process of revision, course 
materials are reviewed and modified; if a task analysis is per- 
formed, material that is no longer relevant will be dropped and 
new material may be added. The result is that the course materials 
used with a new method of instruction are rarely identical to 
those used in the old course; note that the revised course 
materials could be taught by any method of instruction, including 
conventional Instruction. Figure l describes the steps involved 
in changing a course from conventional to computer-based or any 
other form of instruction: the course materials are revised and 
restructured into an appropriate format. Each step in this 
process implies an expenditure of resources; each method of in- 
struction implies a cliff erent final cost, and may yield different 
levels of ins"t rut clonal effectiveness. 

Thus, all comparisons of computer-based instruction with 
conventional instruction produce results (generally student 
time savings) that may be due to the new method of instruction 
and/or to the process of course revision (which may also shorten 
or lengthen the course). There is one study where a computer- 
based course was compared indirectly to its revised, conventional 
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version; all other comparisons are with the original conventional 
course. There are some data where a computer-based course was 
compared to its revised, individualized version. 



CONVENTIONAL 
INSTRUCTION 



INDIVIDUALIZED 
COURSE 
MATERIALS 



REVISED 
COURSE 
AAATERIALS 




-•^CONVENTIONAL INSTRUCTION 



INDIVIDUALIZED INSTRUCTION 



COMPUTER-ASSISTED INSTRUCTION 



COMPUTER-AAANAGED INSTRUCTION 



BASELINE 
COURSE 



EXPERIMENTAL 
COURSE 



1-11-TMf 



FIGURE 1. Steps involved in modifying a course from 

conventional to individualized or computer- 
based instruction. 



C. SUMMARY 

The data base used to evaluate the cost-effectiveness of 
computer-assisted and computer-managed instruction in military 
training has the following limitations: 

• Thirty studies, conducted since 1968, provide ^8 data 
sets on the effectiveness and 8 data sets on the costs 
of computer-based instruction. About half of the studies 
are based on 10 or less hours of instruction; about half 
of the studies are based on 50 or less students; c?. few 
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studies (on computer-managed Instruction) Involve longer 
courses (2 to 10 months) and larger numbers of students 
(600 to 2500). 

The most relevant measure of effectiveness of Instruction 
Is the performance of graduates of courses on the Job In 
an operational unit. Such data are not now available. 
All studies use student achievement at school as a meas- 
ure of effectiveness. The relation between student 
achievement at school and performance on the job has not 
been demonstrated. Some measures of effectiveness that 
have been used (e.g., stuaent time saved and student at- 
trition In courses) should be treated as measures of 
cost . 

The cost data derived from these studies are generally 
Incomplete; the cost data reported In experiments do not 
extrapolate readily to operational settings because of 
major differences In training organizations and accounting 
procedures. No data are provided that permit comparisons 
between the costs of computer-based and conventional In- 
struction. 

None of the studies provide an explicit distribution of 
costs over some specified life cycle for comparable 
methods of Instruction; none provide a model for use In 
estimating costs. 

Most comparisons of student achievement with computer- 
based and conventional Instruction provide confounded 
results that may be attributed either to the method 
of Instruction and/or to revision of course materials. 




III. EFFECTIVENESS OF COMPUTER-BASED INSTRUCTION 



Military training is intended to provide the skills and 
knowledge required to perform various tasks in operational units. 
Thus, the effectiveness of computer-based instruction for teach- 
ing a particular course should be compared to that of conven- 
tional instruction by measuring how well graduates taught either 
way perform the same tasks in the field. Such data were not 
found in the research literature dealing with the effectiveness 
of computer-based and conventional instruction in military train- 
ing. 

Instead, we found that the following measures of effective- 
ness have been used: 

• The amount and/or quality of information and skills 
acquired by students at school (end-of-course achieve- 
ment) 

• The amount of time required by students to complete a 
course (student time savings) 

• The number of students who do not complete a course for 
academic reasons (academic attrition) 

• Attitudes of students (acceptability of computer-based 
instruction to students) 

• Attitudes of instructors (acceptability of computer- 
based instruction to instructors). 

These measures can be collected conveniently at schools or 
experimental sites before students scatter to other assignments 
but they are not necessarily appropriate for evaluating the ef- 
fectiveness of computer-based instruction. Various limitations 
of these measures were discussed in Chapter II. 
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Here the effectiveness of CAI and CMI is considered on the 
basis of the evidence provided by military research studies and 
with explicit recognition of some major limitations to this evi- 
dence, as follows: (l) measures of student achievement at 
school must be validated by data on performance on the Job and 
(2) measures of time saved by students at school and measures 
of academic attrition at school should be treated as measures 
of cost rather than of effectiveness. 

In general, the military interest in CAI and CMI is based 
on the premise that these methods. of instruction may save stu- 
dent training time with little, if any, loss in student achieve- 
ment. The interest of schools and colleges is based on the pre- 
mise that CAI may provide the same or greater student achievement 
than that provided by conventional instruction; there is much 
less concern here for the amounts of time spent by students 
under various methods of instruction. Schools and colleges 
have shown little interest in CMI. A brief summary of the find- 
ings on the effectiveness of CAI for instruction in schools and 
colleges, based primarily on evaluations of PLATO IV and TICCIT 
in community colleges, appears in Appendix C. 

A. SOURCES OF INFORMATION 

CAI and CMI instruction have been evaluated in about 30 
studies conducted by the military Services over the period of 
1968 to 1978. These studies are summarized in a series of 
tables in Appendix D. These studies sample a wide variety of 
courses in technical training, e.g-., basic electronics, elec- 
tricity, vehicle repair, inventory management, fire control, 
and precision equipment, among others (see Table The courses 

include cognitive skills (knowledge, theory, and rules) and per- 
formance-oriented skills (hands-on maintenance, checkout, and 
repair) at a wide range of skill levels. There is no overlap 
between the courses used in evaluations of CAI and CMI. 
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TABLE 4. COURSES USED IN VARIOUS STUDIES OF CAI AND CMI 





No. of 

Evaluations 


Courses 


CAI 


CMi 


DaSIC eWCiiOlliCS 


15 




cwctncity 


5 




maCniniSi 


2 




TraiRmg maienats deveiapnient 


1 




nccipe conversion 


2 




Aircran panel operation 






mraivai assisisni 






venicie repair 


4 




UfMthar 

vf caincr 


1 




Tflctirfll riliinllniitnr 

lOlffUMI bUUIUIIIillUl ^O'wftj 






Rre control technician 






Aviation famlHarlzation 




2 


Aviation mechanical fundamentals 




2 


Inventory management 




1 


Materiel facilities 




1 


Precision measuring equipment 




1 


Weapons mechanic 




1 


Total 


40 


& 
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B. STUDENT ACHIEVEMENT 

The effectiveness of CAI and CMI as methods of instruction, 
compared to conventional or individualized instruction, has 
been measured only by performance of students on tests admin- 
istered at schools, rather than by performance on the Job after 
graduation. Student achievement at school might predict 
quality of performance on the Job but correlations between 
these two measures have not been established for conventional 
or for computer-based instruction. The Services evaluate some 
courses by means of supervisors' ratings of the performance of 
graduates on the Job; however, these results are qualitative 
in nature and have not been collected systematically. Data on 
student achievement at school, found in various studies, are 
summarized in Table 5. 

In 40 comparisons, student achievement with CAI was about 
the same as with conventional instruction in 24 cases, superior 
in 15, and inferior in one. The differences in performance, 
although statistically significant, were Judged not to have 
practical significance. In eight comparisons oT CMI with con- 
ventional instruction, no significant differences were found 
in student achievement at school. 

In addition to these results, there were five cases where 
student achievement on CAI was compared to that on individual- 
ized instruction. Achievement was the same in four cases, 
and superior with CAI in one. 

The fact that student achievement with CAI and CMI is 
about the same as that with conventional instruction or individ- 
ualized instruction is also a direct consequence of the fact 
that students instructed by CAI and CMI are held in these 
courses until they master all lessons. The critical variable 
thus becomes the amount of time needed to complete courses 
given by computer-based instruction. 
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TABLE 5. STUDENT ACHIEVEMENT AT SCHOOL FOR CAI AND CMI, COMPARED 
TO CONVENTIONAL INSTRUCTION, IN MILITARY TRAINING 











HETNOO OF 










STUerr ACNEVafifT AT SCHOOL 


TYPE OF 
TNAMM6 




BnnurnoN 


SYSTEM 


savcc 


LKATMM 














MFenN 


SAME 


SUPBMM 


NEFERENCES 


CAI 


BM1B0Q 

PUTOIV 


A 
N 

A 
N 
N 
N 

AF 
AF 


86NALCAS 
SAN HEM 

mam 

SANOnO 
SANOESO 
NOITM BLAND 

CHANUn 






• • • • 

• • • 

• • 

• • 

o • • • 


• • 

• • • • 
• 

• • 


ELECTNOMCS 
ELECIMOIV 

MACHMBT 

EIKTNOMCS 

AfXrECONVERSMNI 

A/CPANaOTBUTON 

MEDCALASS8TANT 

VBKUREPAJR 


BM (1HI). imm 1S72) QM Mrf irngt (1971) 

FM t^Milli (1170), HatM i L«kn (1171. 1172). Ftt^ SMq» tt^ 

U.S. Amy OiteM CmMt airi ScM (1I7S) 

SMw (1179). Utey. CitwM tid (1979). SMigh Mrf Ctitfy (w^iM.) 

Ffitelcfcs Mrf HNw4Hct (1977) 

CrtvM, HHtKk tid (1979) 

TMIifcirefciir. DrifM ifiC (1977), C9l|UR mM Oukm (1977) 
DMMa,DiLM ir^(19n) 




LTM 


AF 
AF 








* • 


• 


acCTNONSS 
WEATHBI 


Hantt. BiMStafi tid (1972). KnsIw AFB (1972, 1973) 
Dtwt, JlMiHur^ (1972) 




Ticcrr 


N 
N 


NOmtBUND 


• 








TACTCAL cam. (S^) 
FMCONTNOLTGCNMOAN 


WMnr (197S) 

6«Hnl EMte (M«MKt SyttMn (1978). NaMmi Mrf CiauM (1978) 




PUTQIV 


N 


DAMNKX 






• • 




FMECONTNOLTECHNDAN 


OMml SkKtrtc Oi*Mce SytMm (1976). NatfskM Mi Sntsai (1978) 










TOTAL 1 


24 


IS 






NAVY CM 


N 
N 


Moms 






• • 

• • 




AVIATRM FAMUAMZATKM 
AV. MECa FUNDAMENTALS 


C«i«.6nkMi tid (1978) 
Cma.fifalMiiri£ (1978) 


CM 


AB 


AF 
AF 
AF 
AF 


LOWNY 
LOWNY 
LOWNY 
LOWNY 










MVBITONYMSMT, 
MATBOa FAOLmES 
PNEC. MEASUNM6 EQPr. 
WEAPONS MOHAMC 


Brtrtug (1971) 
Bril9ii9 (1978) 
SrMtai (1971) 
9riiliii (1971) 










TOTAL 0 


1 


0 
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C. STUDENT TIME SAVINGS 

Most studies take the amount of student time saved to com- 
plete courses given by CAI or TMI, In comparison to convention- 
al Instruction, as a measure of effectiveness. As pointed out 
above, the amount of student ^ime saved is a measure of cost. 
Student time savings reported In 30 studies are shown In Table 6 
and summarized In Table 7. 

When the findings for CAI and CMI are combined, computer- 
based Instruction appears to sa^e about one-third of the time 
required by students to complete the same courses when given by 
conventional Instruction. However, there Is a wide variation In 
the amounts of savings that have been reported. The amounts of 
student time saved by CAI and CMI cannot be compared because In 
• no case was the same course given by both methods of Instruction. 
Two major uncontrolled variables In these studies are the un- 
known quality of the Instructional materials used In the various 
comparisons and uncertainty that the same amounts of course 
materials were used In both methods of Instruction. This argues 
against trying to Interpret apparent differences In the amounts 
of student time saved by CAI or CMI, or by different courses, 
and so on. ' 

There are three Instances where the use of CAI Increased 
rather than decreased student training time and one where Its 
effect was zero. These may be attributed to Inadequate prepara- 
tion of course materials or other factors not explained In these 
experiments. These atypical results occur only In some Initial 
studies and not In more recent ones; In any case, such findings 
would not be recommended for operational use. 

The fact that CAI and CMI save student training time is 
consistent with well-known Information about wide differences 
In student ability (as represented In the normal distribution 
curve) and In the amounts of relevant knowledge held by students 
at the start of any course. In conventional Instruction with a 
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TABLE 6. AMOUNT OF STUDENT TIME SAVED IN COURSES GIVEN BY CAI AND CMI 
COMPARED TO CONVENTIONAL INSTRUCTION. IN MILITARY TRAINING 



•YSTBI 



lavcs 



LOCATVM 



TmOF 



CM 



■MIHt 

LTS^ 

vocn 

PUTOIV 



A 
N 

A 
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N 
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Af 

Af 
Af 

N 

N 

N 
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IS 



ELKTMMCS 
aECTWTY 
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19 
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TABLE 7. AMOUNTS OF STUDENT TRAINING TIME SAVED 
BY CAI AND CMI, COMPARED TO CONVENTIONAL INSTRUCTION 



Method of 
Instruction 


Number of 
Comparisons 


Student time savings, 
compared to conventional 
instruction, percent 






Median 


Range 


CAI 


40 


29 


-31 to 89 


CMI 


8 


44 


12 to 69 


Combined 


48 


32 


-31 to 89 



fixed amount of time, these differences lead to variations in 
the amounts of knowledge acquired by the end of the course, 
i.e.^ as shown by a distribution of final grades. In individual- 
ized instruction, whether computer-based or not, each student 
proceeds at his own pace and differences between students in- 
fluence the amounts of time they need to complete the course 
more than it does the amounts of information acquired. Most 
of the time savings in individualized instruction are produced 
by those students for whom the rate of progress set in conven- 
tional instruction would be too slow; typically that rate might 
be one that permits about 90 percent of the students to complete 
the course during the fixed period of time. 

Almost all of the data shown in Table 6 represent time 
savings found in experiments or operational tests over short 
time periods and with limited numbers of students. Figure 2 
shows the amounts of time required by about 11,000 students to 
complete four courses on the Air Force Advanced Instructional 
System (AIS), Lowry AFB over 2k months ending September 1978. 
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FIGURE 2. Days Required to Complete Four Courses 
on Air Force Advanced Instructional 
System, Lowry AFB, October 1976- 
September 1978 
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It Is clear that the initial savings, such as might be reported 
In an experiment, are maintained over time and, despite monthly 
fluctuations, tend to Increase. The reasons for these reduc- 
tions and fluctuations have not been explored; they could be 
due, at least in part, to periodic revisions in the courses 
(Indicated on the figure), to Improved control over the new 
method of Instruction, to variations In student aptitude and 
to turnover among Instructors. Similar reductions In student 
time are shown In Pig. 3 for about 12,000 graduates In three 
courses on the Navy Computer Managed Instruction System at 
Naval Air Technical Training Center^ Mllllngton, Tennessee, 
over a 15-month period ending May 1978. No significant changes 
were made in these courses during this period. 

D. STUDENT ATTRITION 

Since the method of Instruction may Influence the number 
of students who can successfully complete a course, the rate of 
academic attrition associated with alternative methods of in- 
struction Is a matter of concern. As noted previously, the rate 
of attrition is a measure of the cost of Instruction since It 
Influences the number of students needed to enter a course In 
order to produce a specified number of graduates. Attrition 
for nonacademlc reasons, such as for medical or disciplinary 
reasons. Is not considered here. It should also be recognized 
that the rate of attrition observed in a course may be Influenced 
from time to time by policy decisions on standards for recruit- 
ment and the number of graduates to be produced by various 
courses. Such Influences, if present, are not addressed here. 

Meaningful data on student attrition related to computer- 
based Instruction should come from steady-state applications 
and not from short-term experiments. This condition is met 
marginally by the Air Force Advanced Instructional System (AIS), 
where four courses were Increasingly linplemented on a computer- 
managed Instructional system over the period of 1974 to 1978 and 
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FIGURE 3. Time (Student Contact Hours) Required to Complete 
Three Courses in Propulsion Engineering Basics on 
the Navy Computer Managed Instruction System, 
Millington, Tennessee, March 1977 to May 1978 
(Students at Navy Training Center, Great Lakes, 
Illinois) 
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by the Navy Computer Managed Instruction System, where data are 
available on seven courses before and after Implementation In 
March 1977. 

Figure li shows that, compared to previous rates, academic 
attrition may have Increased In the four courses Implemented on 
AIS. Note, however, that academic attrition appeared to rise 
In all (non-AIS) courses at Lowry AFB over the same period; 
thus. It Is not obvious that the Increased attrition In the AIS 
courses should be attributed primarily to the Introduction of 
CMI Instruction. 

Figure 5 shows academic attrition for seven courses be- 
fore and after Implementation on the Navy Computer Managed In- 
struction System. The average rate of academic attrition In 
these courses was 3.2 percent before and 4.6 percent after Im- 
plementation on the Navy CMI system (It increased In six 
courses and decreased In one). Data on comparable courses not 
on CMI during the same period were not provided. 

Little data are available on academic attrition during 
experiments. Longo (1972) says that academic attrition was 
about the same for two courses In basic electronics taught by 
CAI or by conventional Instruction; Gluntl and Longo (1971b) 
say that attrition was 22 percent lower for the CAI group In 
another study; there were few students (66 - l86) In any of the 
studies summarized here. The use of CAI on four Special Pur- 
pose Vehicle Repairman courses at Chanute AFB produced no 
significant effect on academic attrition over a 9-month period 
(Dallman, DeLeo, Main, and Glllman, 1977); about 300 students 
were Involved. Initial results for four courses on the Navy 
CMI system In 1975 suggested that there were no effects on 
student attrition at that time (Carson, Graham, Harding, et at., 
1975). 
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FIGURE 4. Academic Elimination Rate in Four Courses Before 
and After Implementation on AIS at Lowry AFB 
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FIGURE 5. Academic Attrition In Seven Courses Before and 
After Implementation on Navy Computer Managed 
Instruction System 
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In summary, only two CMI systems, the Air Force AIS and 
Navy CMI, have received extended, though still limited, use 
in military training. Academic attrition may have increased i^/^ 
courses taught this way, compared to attrition with conventiot^Al 
instruction during prior periods. Since these comparisons do 
not take into account possible changes in the qualifications 
students over the same time periods, the available data sug- 
gest but do not prove that CMI may increase academic attritior^ 
over that found with conventional instruction. 

E. ATTITUDES OF STUDENTS AND INSTRUCTORS 

Attitudes of students and instructors to CAI or CMI in 
military training, compared to conventional instruction, are 
noted here only as qualitative aspects of these methods of in% 
struction. Most of the data came from experiments of short 
duration. Data on student attitudes towards CAI or CMI are f^i^nd 
in 39 of the ^0 reports summarized in Appendix D. As shown 1^ 
Table 8, students almost always favor CAI or CMI over conven- 
tional instruction, or at least say so when asked; they are ur\A 
favorable to CAI in one case and find no difference in another^/ 

TABLE 8. ATTITUDES OF STUDENTS AND INSTRUCTORS COMPARING, 
CAI OR CMI TO CONVENTIONAL INSTRUCTION IN MILITARY TRAININGV^) 



Attitude to CAI/CHH 


Students 


hittructors 


CAI 


CIM 


CAI 


CMI 


Favorable 


29 




8 


1 




No difference 


1 










Unfavorabie 


1 






4 


4(b) 


No report 


1 






27 


4 


Total 


32 


8 


32 


8 



^''AS dtta an nttmbir •( raports ttimiiiariztd lii Apptodto D. 
^^FivoraUi to CM it first, changing to untavarabii by and 9( ftudy. 

12-2f-7S-11 
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Instructors' attitudes are reported only In 9 of these Ho 
comparisons; instructors are unfavorable to CAI or CMI in 8 of 
these 9 cases and favorable to CAI only in 1. 

Instructors of courses taught by CAI or CMI have not re- 
ceived much attention by researchers. According to two studies 
still in draft (February 1979), only half of 5^ instructors 
sampled in 1977 at the Naval Air Technical Training Center, 
Millington, Tennessee, believe that individualized instruction 
is as effective as conventional instruction.* The training of 
instructors is still oriented largely towards conventional in- 
struction, and instructors assigned to CMI receive little guid- 
ance on how to conduct such courses. 

F. COMPARISON OF TIME SAVINGS FOUND WITH INDIVIDUALIZED 
INSTRUCTION AND COMPUTER-BASED INSTRUCTION 

Student training time in courses can be reduced without 
resort to computer-based instruction, e.g., by reducing the 
amount of material to be mastered in courses, increased re- 
Uance upon on-the-job training, improved conventional instruc- 
tion, and by individualized instruction (which, by definition, 
excludes computer support). It is far beyond the scope of this 
paper to consider all of these possibilities. However, we found 
some data on the amount of student time saved when the same 
courses are given by individualized instruction and by computer- 
assisted or computer-managed instruction, compared in all cases 



^'Practical problems in the implementation of individualized 
instruction. Navy Personnel Research and Development Center. 
San Diego, California (draft). « c , 

Instructors' attitudes towards computer-managed instruction 
Navy Personnel Research and Development Center, San Diego. * 
California (draft). * 
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to conventional Instruction. The essential Issue concerns the 
benefit. In terms of additional student time saved, when com- 
puter support Is given to Individualized Instruction (without 
computer support). Data on 12 courses are summarized In Table 
9 and shown In Pig. 6. 

TABLE 9. AVERAGE AMOUNT OF STUDENT TIME SAVED BY 
INDIVIDUALIZED INSTRUCTION AND CAI OR CMI IN THE 
SAME COURSES, COMPARED TO CONVENTIONAL INSTRUCTION 



No. of 


Average Amount of Student Time Saved 


Courses 


individualized 
instruction 


CAi 


CiMI 


5 


64% 


69% 




7 


51% 




51% 



12-2t-78-12 3-26-79 



Individualized Instruction saves large amounts of student 
time otherwise required by conventional instruction (average 
savings of 50 percent or more in these samples). The addition 
of CAI to five individualized courses produced additional aver- 
age time savings of 5 percent; the addition of CMI to seven 
courses produced no additional time savings. Again, no signifi- 
cance can be given to the differential time savings observed by 
adding CAI or CMI to individualized instruction because different 
courses were used in each comparison. 

These data do not necessarily imply that the addition of 
computer support to individualized instruction is not cost- 
effective. That would depend, in each case, on whether the in- 
cremental costs of computer support are offset by cost reductions 
in other areas, such as for the number of instructors and support 
personnel, administrative services, and other factors. 
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Little attention has been given to the benefits associated 
with different types of computer-based support of individualized 
instruction. For example, from early AIS data. Student Progress 
Management saved an average of 9 percent of student time in four 
courses and Individualized Instructional Assignment saved an 
additional 3 percent in one course and none in three others 
(McDonnell Douglas Astronautics 1977a). Only student time sav- 
ings were considered in these T-gports and no attention seems to 
have been given to other possible benefits of computer-support 
to instruction. 

6. SUMMARY 

1 • Effectiveness of CAI and CMI 

The effectiveness of CAI and CMI has been evaluated in 
many different types of courses in military training, e.g. y 
electronics, vehicle repair, and inventory management. These 
courses include both knowledge and performance-oriented skills. 
The effectiveness of CAI and CMI cannot be compared directly 
because in no case was the same course given by both of these 
methods of instruction. 

2. Student Achievement 

Student achievement at school is about the same for CAI, 
CMI, and conventional instruction. Some evaluations show that 
student achievement with CAI is superior to that with conven- 
tional instruction but these differences are Judged not to have 
practical significance. 

3- Time Savings; Computer-based Instruction vs. Conventional 
Instruction ^ 

Computer-based instruction appears to save about one-third 
of the time required by students to complete courses given by 
conventional instruction. There is a wide variation in the 
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amounts of time reported as saved In experiments but research 
has not addressed this issue. Based n experience gained in the 
AIS and Navy CMI systems, th-i amounts of student time saved ap- 
pears to increase ovei time; the amounts of student time saved 
also fluctuates from month to month. 

4- Time Savings: Computer-based instruction vs. Ind ividualized 
Instruct! on ~ 

About the same amounts of student time are saved when the 
same courses are given by individualized instruction without 
computer support or by CAI or by CMI. 

5. Student Attrition 

Student attrition for academic reasons appears to increase 
slightly when CMI replaces conventional instruction, based on 
experience with che AIS and Navy CMI systems. The possibility 
that these increases may be due, ±n part, to changes in student 
quality and to other factors has no^ been examined." 

6. Student and Instructor Attitudes 

Student attitudes to CAI and CMI tend to be favorable. 
The attitudes of instructors appear unfavorable to CAI and CMI 
in comparison to conventional instruction, but this finding is 
based on limited data. The role of instructors in CAI and CMI 
has received little attention, both as to collecting more re- 
liable information about their attitudes and to preparing them 
for handling these new methods of instruction. 



IV. COSTS OF MILITARY INSTRUCTION 



Surprisingly little appears to be known about either the 
determinants or composition of the costs of Instruction, Each 
Service has a system for reporting the costs of training courses 
for such purposes as development of estimating relationships and 
evaluation of proposed training program alternatives; their 
actual use appears to be limited to providing the average total 
(bottom-line) costs of Individual courses for such purposes as 
setting reimbursement rates for training foreign students and 
those from other Services. This Information sheds no light on 
questions of why training costs are what they are or how they 
would change In response to changes In training courses, such as 
the method of Instruction or the content of course materials. 
In general. It may be said that neither the detailed data on 
training costs nor the methodology for analysis (as opposed to 
accounting) of training costs have been developed. The develop- 
ment and maintenance of a data base on the costs of military 
Instruction are far from cost-free, and the question of what data 
"should" be collected can only be assessed by further questioning 
their cost and worth of such data In supporting cost-effective- 
ness analyses of military training. 

This chapter addresses two problems associated with the 
collection *of data on the costs of training. The first Is to 
examine how the organizational structure of formal mllltax'y 
training affects the collection of relevant cost data. The 
second Is to assess data on costs of training that have been 
developed In various experimental 'programs and studies of com- 
puter-based Instruction. 
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A. COLLECTION OF COST DATA 

The resources required for Instruction are structured simi- 
larly in both military training and civilian education. For 
example, consider a school which offers a few courses in a limi- 
ted number of subjects and grades. Resources can be grouped 
according to type or function. They can also be grouped accord- 
ing to where, within the organizational structure, these resources 
are directly applied and identified. Thlo two-way grouping de- 
scribes a matrix, as shown In Pig. 7. Some resources, such as 
for instructors and certain equipment, are dedicated to a par- 
ticular course and their costs can be associated directly with 
that course. Other resources, such as for facilities and other 
equipments, serve a number of courses in common. Requirements 
for the use of common resources may vary widely between the 
courses offered, with the result that the costs of different 
courses may vary significantly in ways that are concealed. 




Casts IdmtHM only in open eels 
*Tnnsporta1ioa, cifetnti, etc. 



FIGURE 7. Resource matrix 



60 

73 



1 . Organization of Military Training 



At a small school, the use of common resources by particular 
courses are readily visible and thus their costs may be easily 
associated with different courses. For example, the cost of 
using media devices may be allocated systematically to indi- 
vidual courses by keeping simple records of purchase and main- 
tenance costs and the hours of use in different courses. In 
current military training, the relationships between courses, 
resource use, and costs of training are complex and obscured by 
the large size of training establishments and the manner in which 
they are organized. Some types of resources are expensive to use 
compared with others, and some courses are expensive to offer 
compared with others; the records required to trace the use of 
resources by particular courses are, themselves, complex and 
expensive. This is the heart of the problem faced in the collec- 
tion and evaluation of data on the costs of training. 

A representative organizational structure for military 
training is shown in Pig. 8. It is adapted from a "typical" 
U.S. Air Force organization as shown in Hess and Kantar (1977). 
The critical feature is the deep hierarchy of organizations that 
support and manage individual courses and lessons. An explana- 
tion for this degree of complexity is that it is needed to attain 
an efficient scale of operation. For exajnple, instructional ma- 
terial is typically associated with an individual lesson and 
instructors may be best utilized as specialists in a single 
course. However, some training resources, especially in tech- 
nical training, come in large and indivisible units thac may be 
employed efficiently only where student loadings exceed those of 
individual courses, training branches, departments, schools, and 
possibly the total student population of a training facility. A 
conspicuous example of a large and indivisible training input is 
the central processor of the PLATO IV system. 
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The formal organizational structure extends through the 
training branches that are responsible for a number of closely 
related courses. The course '(and ultimately the individual 
lesson) is the elemental unit of training and the generator of 
all training costs; resource expenditures necessary for its con- 
duct occur at each of the higher levels of the structure (includ- 
ing the training branches), and the expenditures incurred at any 
higher level node may support the whole range of instruction ana 
support activities beneath it. In essence, the whole structure 
above the course level is equivalent to the columns labelled ^'^all 
courses" in a small school (Fig. 7). In a large training 
organization, the visibility of who provides what for whom and 
who receives what from where is quickly lost. The relationships 
among units at various levels in military training organizations 
are too complex to be traced by simple bookkeeping procedures. 

In the absence of extensive data, the relationships between 
ir itruction and the expenditure of resources for instructions! 

upport in military training cannot be determined. The C33t3 of 
supporting particular courses can be estimated only by hig^v/iy 
arbitrary allocation rules that may bear little resemblance to 
the true sources of cost. Discovering these relationships is an 
essential ingredient of the capability to evaluate training costs. 

2 . Alternative Ways of Collecting Cost Data 

Mc . c-: , /-ecise and detailed cost data are needed to support 
cost-effectiveness analyses of alternative methods of instr-.ction 
in military training. Training cost data can be collected in 
three ways that differ widely In scope of effort and Cv?st. The 
first is to collect cost data> through a formal reporting system, 
for all training courses offered by a Service, including all 
costs for conducting and supporting training programs. The ex- 
tent, detail:> and identification of cost data would have to be 
greater than that provided by current systems, especially with 
regard to training support functions. Since a course may employ 
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several methods of instruction (conventional, individualized, 
and computer-based), data would have to be identified with small 
units of instruction, possibly individual lessons, if the system 
were to provide Information on the relative costs of different 
methods of instruction. 

The second method is to collect information, also through 
a formal reporting system, for a sample of organizations that 
conduct and support training, it is recognized that the military 
Services have rarely applied sampling techniques to collect cost 
data. Basic questions regarding the extent and duration of samp- 
ling would have to be resolved before a program for collecting 
data could be designed or procedural problems addressed. The 
collection system could not disturb existing management and 
data-collection systems, and it would have to be implemented in 
a manner that would not distort either the level or structure of 
training costs in the activities to be observed. 

' The third alternative is to perform ad hoc studies of 
organizations that conduct and support training programs. Costs 
of ad hoc study should be lower than those of formal reporting 
systems, and it offers the advantage of flexibility. Studies 
can address specx.'ic topics of high interest and focus attention 
on cost and non-cost responses to systematic changes in study 
parameters, e.g., student .-instructor ratios or length of train- 
ing day; data collection may be tailored to the questions ad- 
dressed and organizations examined. Ad hoc study prog-ams also 
have problems. A principal one is to maintain financial support 
that is adequate for pursuing a coherent and on-going program 
and for providing data in a timely fashion and depth that recog- 
nizes the full extent and sources of all training costs. 

B. AVAILABLE DATA ON COSTS OF INSTRUCTION 

This section summarizes the' data we were able to find on 
the costs of computer-based and other methods of instruction; 
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the detailed data are presented in Appendix E. Here, we discuss 
the adequacy of these data for assessing the costs of different 
methods of instruction. We have excluded data that were not 
described well enough to be interpreted with confidence. All 
data are shown as they were found in the literature. No investi- 
gations of their validity have been performed, and no adjustments 
have been made for changes in price levels. 

Table 10 shows the number of sources of data on costs of 
instruction, arranged according to method of instruction and type 
of resource. The cost data come either from eight experiments 
(identified in Table 1, p. 27) or from other sources (identified 
in Appendix E) j shaded cells indicate that cost data are not 
applicable; blank cells indicate that relevant cost data are 
not available. 

Table 10 shows that there are few sources of data on the 
costs of instruction in military training; especially notable is 
the absence of information on the costs of conventional instruc- 
tion. The troublesome nature of this is obvious when one con- 
siders that the cost-effectiveness of CAI or CMI must be compared 
to that of some other method of instruction, generally conven- 
tional instruction. Two other important omissions concern Program 
Design and Program Management. The design of instructional pro- • 
grams (Program Design) may be a significant cost item because of 
current emphasis on Instructional System Development in all 
military Services, but its costs appear to be either ignored or 
combined with those of instructional materials. Since Program 
Design is independent of the method of instruction and may have 
its own impact on instructional costs, its cost should be sepa- 
rated from the cost of instructional materials. Program Manage- 
ment may be a major cost item because of the large and highly 
sti?uctured organizations in which military instruction takes 
place, and this cost may differ between instructional methods. 
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TABLE 10. NUWER OF SOURCES OF DATA ON COST OF INSTRUCTION. 
ACCORDING TO METHOD OF INSTRUCTION AND 
RESOURCE TYPE OR FUNCTION (See Note) 



RESOURCE (TYPE OR FUNCTION) 




Program Otvoltpnunt 
Program Ootign 

tostnictlMial MaMals:^vtiitfoMi BMlructtaii 
MhridMlzod bittnictlon 
Programming 
Hret m PreductlonC 
Computor-Bssod Instmctfon 



Program Doflvory 
Bnttnictioo: tostrvctore ' 

iRstnictioflal Sopfmrl Pononnol'' 
Equipmoiit and Strvicos:* Latowatory (ind. simulalort) 
Mo«aDovtcas 
ComiHilor Syttoms 
Communications 
Matoflals (Ind. Consumables)' 
FacWlosfl 

Program Managomcnt and Administration 
Studoflt Porsonnol: Pay and Alowancos 

Ottiora(PCS, TOY, 01) * 



NOTE: tmnm i ppl cHH . IM c«» liteti tM rstovM ettt tou art wt tviiiiM 

%CMCS TCOr. BM 1900. LTS.3. BEH. Mi M $ipmkMM iMpbMii tyttM. 



'AI *fCt pMMMMl Ml IkMM hi lONr MliOiflM. 

■ov ^ n«r 1M1, MMpmr sytMi MOwtrt; Mb 
VcMm MplM tr hstnidM MtMWs (Mu. cMrstwara m^Im. fti). 
IStractam, txlvtt. Ml IMsWais. 

^f"^ ''"II""' *» ipi*ci«i It H mttMs K tottmctiM (ixctft wbm stamtaM In CAI systMul tod tbtn 
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The sample of data on costs of training presented here is 
meager by any standards. We cannot begin to explain the range 
of costs that has been observed, and we feel uncertain as to the 
feasibility of a generalized parametric approach to estimating 
the costs of training. Extensive further efforts to collect and 
interpret data on the costs of training would be required before 
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the value of parametric analysis of training costs could be 
Judged or a general model could be formulated. 

The following sections discuss. ^th6 cost data that we vi^>re 
able to find and use. 

1 . Program Development: Program Design 

No data were found on these costs of instruction. 

2. Program Development: Instructional Materials for 
Conventional Inst rue ti"orr 

No data were found on these costs of instruction. 

3. Program Development: Instructional Materials for 
Individualized Instructfon" 

Development of instructional material for individualizi^'^ 
instruction encompasses two distinct and separable function^ ^ 
the programming (authorship) and the production of first un:i^s 
(master copy) of media material (including printed text), 
cost data we found are shown in Table 11. The most notable 
feature of resource requirements is the great variation ass^/^ 
ciated with both functions. 

With regard to programming (authorship), the available In- 
formation shows two widely separated values; i.e., ko and 2?3C) 
man-hours per instructional hour; the source provides no dii^A- 
cussion for the large difference in values for the two medi^\ 

With respect to first unit production , master co^^) , 
costs of different media range from $12,000 for an hour of i^/^und 
motion picture or TV tape to a few hundred dollars for primt^d 
text and silent slide or film-strip (assui.ilng 30 pages or f^Ames 
per hour). For the same media, costs ra ^ » from near $500 \^ 
$2,000 per instructional hour for sound-sliOs and $10 to $4(t)C> 
for printed illustration, and the literature provides no exj^^a- 
nation for these differences. 
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TABLE n. INDIVIDUALIZED INSTRUCTION: INSTRUCTIONAL 
MATERIALS DEVELOPMENT, REQUIREMENTS, AND COSTS 



RESOURCE 


IMT OF MEASURE 


NUMBER OF 
SOURCES 
OF DATA 


ESTMATU 


REFERENCES 


SMMi IMm Pteturt r TV 


Mio hoiiff /hiliiif thw^* Hmit 
Mao-koort/bistnicttOMl Hoor 


1 
1 


280 
40 


U.S. Army OriMmco Cootor aiML 

SdMol (1975) 
U.S. Army OrdooAco CoMor and 

ScHool (1975) 


SMiid IMm PIctHTt or TV 

SRMt MoIIm PIctiirf 
Sltnt SWt (ir Hkn Stri^) 


Dolars/infftraeflooaj Hoor 
Dolart/littnictfoul Hour 
Dolin/hittractfooal Hour 


2 
2* 
1 


$8,000-12,000 
$ 500- 1,925 
$ 10.200 


Hau aad Kaatar (1977); U.S. Army 
Ordaanea Caatar and ScM (1975) 

Hau aad Kaatar (1977); U.S. Army 
Ordaaaca Caatar and Scliool (1975) 

Hats aad Kaatar (1977) 


Rtala 
ftittrattM 

PvlnM 


DoRars/sado 
OoRors/Sldo 


1 
1 


S 1 
$ 3-115 


ntts aao Kaatar (1977) 
Hats aad Kaatar (1977) 


Ttxt 

■uttntfan 
Awto 


DoRsrs/Pigo 
Oolart/Pigo 

DoRars/tostniefloaal Hour 


1 
1 
1 


$ 7 
$ 11-430 
$ 180 


Hats aad Kaatar (1977) 
Hast aad Kaatar (1977) 
Hats aad Kaatar (1977) 


CMNbiMd ir Not SptcHM 

l-M-70-31 


MwHioure/lRStnicllonal Hwir 
DoRara/bimcttoMi Hour 


2 
2 


40 - 200 
$1,130-15,800 


Dalaiaa,lteLaaar 0^(1977); 

MMatao. Papottl. aad MtoM (1974) 
Patoyn, BaadlMda, Brakka (1977); 

TamUn, Caaaaly ar^ (1975) 
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With such wide ranges, the usefulness of cost-effectiveness 
analysis may be questioned. One of ns principal applications 
lies in identifying and separating promising from unattractive 
alternatives early in course design, i.e. y before significant 
resourc;>s have been committed. Much of its value in this role 
is lost if a significant course design effort, to identify media 
mixes, is required to provide initial assessments of alterna- 
tives. This situation argues for the application of course 
design procedures such as MODIA '(Carpenter-Huffman, 1977) and 
TECEP (Braby, Henry, et at., 1975) at command levels where rele- 
vant policy is formulated and decisions are made (see Lackland 
APB, 1978). 




4. Program Development; Instructional Materi als for 
computer-based instruction ^ 

In computer-based instruction, development of instructional 
materials also encompasses two distinct functions. The first is 
programming or authorship, similar to individualized instruction. 
The second, coding, organizes the material into a form suitable 
for machine processing. Resource requirements for close to 1,000 
hours of instruction were cited, and large variability is again 
present (see Table 12). 



TABLE 12. COMPUTER-BASED INSTRUCTION: REQUIREMENTS FOR 
INSTRUCTIONAL MATERIALS DEVELOPMENT 



METHOD OF 




MtmucmoNAL 

HOURS 
OEVaOPED 


MiUMIOUin f« MSTIIINmONAL HOUK 




MsmucnoN 


SYrrai 


PMMMMMMO 


C00M6 


TOTAL 




CAI 


PLATO nr 


39.0 

0.0 

2.0 
30.0 
70.0 
32.0 
315.0 


27 te 240 
141 


50 t0 457 
81 


77 ti 714 

155 

400 

204 
100 A 200 

222 
80 


HMtock A Slw|h (1976) 
Kiftt (1976) 
Kite (1976) 

U.S. Anny Ovtatct CmIw A ScM (1975) 
Mmm. OiLte afaL (1977c) 
Hbmlcil(1977) 
Mbms (1975) 




VCCST 


10.0 
3.0 
32.0 


150 


95 


200 
400 
240 






LTS4 


30.0 






175 






BII-19M 


3.5 


350 


119 


475 


llSa99i« A ^TMsfiii (1974) 




UnyicMN CM 


(Maim 






150 4 200 


MUMmi. Pipttt A McM (1974) 


GM 


Navy CM 


50.0 
300.0 
UikMwa 


100 


10 
(25)« 


110 
30 A 60 

(316)« 


C8nM,0raka««rAC(l975) 

HaiSM,nMSif«t(1975) 

Pifieyi,BaidMifrA((l977) 



M iteM li im « Mm ptr ImMmhI IM Mi km feM e«m^ 



For CAI, authoring is cited as ranging between about 30 and 
360 man-hours per Instruction hour, and coding between 50 and 
^70 man-hours. These ranges cannot be attributed to extraneous 
factors, such as differences in the way expenditures are accounted 
for, since close to the total range of variation was noted in one 
study (Hurlock and Slough, 1976) summarizing eight experimental 
programs performed by the same organization and utilizing the 
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game CAI system In roughly the same time period. These data are 
also insufficient for attributing different programming and coding 
requirements to different CAI systems and/or alternative instruc- 
tional strategies. For CMI, the variation is similar; from less 
than 30 to 290 man-hours for authoring and 10 to 25 man-hours for 
coding. 

5- Program Delivery; Instruction, Including Instructors 
and instructional Support Personnel 

Only three sources provide data on costs of instructors and 
instructional support personnel. A few citations to student 
personnel cost are also included here. The data are limited to 
pay and allowance rates, student : instructor ratios, and, in one 
source, instructional and indirect support personnel ratios (see 
Table 13). 

Personnel expenditures are considered to account for the 
bulk of training costs, and those associated with students, in- 
structors, and with instructional, support personnel must be 
aseumed to be significant. In the absence of other changes, 
decreases in course lengths (e.g., associated with an introduc- 
tion of computer-based instruction) would result in lower student 
loads and proportional decreases in instructor personnel. 

6- Program Delivery; Laboratory Equipment 

No data were found on these costs of instruction. 

7- Program Delivery: Media Devices 

Estimating costs of use of media devices entails extensive 
training course specification similar to that associated with 
production of master copies of media materials. A large variety 
of devices is available at widely differing costs. Equipment is 
long-lived and can be shared by different courses. Representa- 
tive cost ranges are shown in Table l4. 

One comprehensive catalogue (The Audio-Visual Equipment Di- 
rectory, 1978, published by .the National Audio-Visual Association) 
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TABLE 13. INSTRUCTOR AND SUPPORT COSTS FOR PROGRAM DELIVERY, 
COMPUTER-BASED INSTRUCTION 





SOURCE 


DATA ITEM 


Carson, 

Graham, 

Harding 

(1975) 


Hansen, 
Ross, 

Bowman. & 

Thurmond 

(1975) 


Crawford, 
Hurlock, 
PadiUo. & 
Sassano 
(1976) 


System 


Navy CMI 


Navy CMI 


PLATO IV 


Cria»{fe in Ratios 
oiiiuBiiis > insiruciors 

Students to Instructional 
(Direct) Support 

Students to Indirect 
(Base) Support'' 


10:1 to 
16:1 


7.5:1 to 
9.0:1 

Unchanged 
at 24:1 

Unchanged 
at 12.5:1 


a 


Pay and AHowance Rates 
Students 
Instructors 
Instructional Support 
Indirect Support 


$5,899 
9.697 


$ 5,300 
10,800 

12,400 


$61,000C 
61,000C 



■Ciniwt bt fxprttMd to ttwM tormt. The net retuH was to eHfninate the single instnictorhour conUinsd In a 9'hour 
traliitog lifintfit 

^Apples to itudtiits. tostructors. and bistructlofial support penonnel. 



CThi $61 .000 figure is descrtbed as t»«et cost and includes a variety of personnel support Hems over and atMve 
pay and aftowancos. cofflmand and adninistntion. dependent school costs, recruiting costs. reenNstment 
bonuses, and reUremont; students and instructors were pilots. 



lists nearly 1,000 presentation and presentation control devices 
classified into over 50 types of commercially available equipment. 
The size of some types of equipment {e.g.^ motion picture pro- 
jectors) varies between that suitable for a large auditorium to 
that used by an individual. The purchase cost of some types of 
equipments will vary by more than an order of magnitude, depend- 
ing upon size and features. Selections of equipments, then, 
require specifications of both the type of device and the en- 
vironment in which course materials will be presented. 
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TABLE 14. MEDIA DEVICES: UNIT COSTS AND OTHER INFORMATION* 



MuM DcT^ 


MTMLCOST 
(DOLLAIIS) 
HEeS AND KANTAR 
(1977) 


1 KC 
UX 

SPAN 
(YEAR) 
HESS AND KANTAR 

(1977) 


MEAN THE BETWEBI FALURES (HOURS) 


HESS A KANTAR 

(1977) 


McDonnell douolas 

11977a) 


ffmni Mtvlf PriiM Im 


17R.1 flflfl 
1 /9*1|UUU 


D 


00*110 


107 


VUMttM nifM^nfriiMiii 


ftflfLA AAA 


9 






3mmI SMt/8frii rritoctMi 


ifliLi Ann 






A9A 


Slnrt Mtvto HitoUMi 










ftigni IMi/fttota Pr^efwi^ 




6*10 


Q0.1S0 

*U* I9U 


177 1 7H 




KAiLO Ann 










•nn 

OU* BUQ 








AvMifla/Dltc Havm 








2,783 












kum viml 










Rjtf Cr^ 


230-1,000 










1950 








Visual 










RattCfiilroi 


,140-380 








Constant Control 


220-1,200 








Audio 










Rata Control 


190-470 








TV Monitor 








2.315 


Hoodsot 








27,240 



HuMtt •vrifMMts IM laiio f9r m li MMteH ctasMMm. Casts an tar CMMtreW qsMty iqiipwuts. 
^cMn ivwiMtf pnfKtws. 



Complicating the problem of determining the cost of use is 
the fact that media devices are typically long-lived and repair- 
able assets, are generally portable, and can be employed in a 
number of classes at a number of locations. The cost of use, 
then, depends upon anticipated lifespan, failure rates, and re- 
pair costs, in addition to purchase cost and rate of usage* The 
cost attributed to an individual course also depends upon whether 
required equipments are currently on hand and available for use 
(i.e., currently unemployed). The latter point implies that one 
must also consider current inventories and usage rates of other 
courses in determining the cost of using media devices. 
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®* Program Delivery; Computer Systems 

Substantive cost information is available on five computer 
hardware systems—IBM 1500, PLATO IV, TICCIT, GETS, and Navy CMI 
These five represent a wide range of capabilities in terms of 
the number of terminals supported by a single central processor. 
They also differ widely in terms of contractual arrangements 
under which they have been obtained (purchase, lease, or a com- 
bination). This makes interpretation of the information impre- 
cise and subject to considerable qualification. 

The detailed information that we were able to compile 
appears in Appendix E. Table 15 summarizes the costs of these 
systems expressed in the following ways: 

• Central processing unit 

• Terminal 

• Total system hardware 

• System cost per terminal 

• System cost per student-hour 

These data should be accepted primarily for illustrative pur- 
poses; any contemplated application would need current data on 
systems configured to particular specifications of interest. 

Three principal resource categories can be associated with 
computer system use: (1) the hardware, (2) its operation, and 
(3) its maintenance. Little information is available on either 
maintenance or operations. Maintenance estimates, based on the 
IBM 1500 and PLATO IV, range from 15 to 35 percent of hardware 
purchase cost over a 5-year period; the lower limit is asso- 
ciated with the IBM 1500. However, all IBM 15OO systems in the 
sample were leased, md the I5 percent figure is based on amor- 
tizing lease charges over a 5-year period. Operating cost may 
vary greatly as a function of the user's organization, and 
sketchy information on IBM 15OO use indicates such a variation, 
ranging between 5 and 50 percent of annual lease costs. 
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TABLE 15. COSTS OF COMPUTER SYSTEMS HARDWARE 



•1 


CMipirtif Syflws 


CmM 
Pncttiir 

(iNisatisi 


Ttmriul. 
IMCMt 


SytttiB 
Hifiwaff 

Ctst 
(TiMiiuiids) 


Syttni 
IMwart CmI 
Hr Ttrniul 
(Tbtuundi) 


SyftMi 
Karrfwart Cati 
Ptr Stutftat- 




BMisao 














32TenMt^ 




— 


$ 600 


$25 


$2.49 




M ATfl III 
nJIIU IT 














1.000 TsnriMb^ 


$5,000 


$5.7 


10.700 


11 


1.46< 


CM 


TKOT 














32Traiuis* 


760 


2.1 


850 


27 


2.66 




MTiraMi 


670 


2.5 


I.UaU 


10 


1.64 




12STtnMs 


970 


2.0 


1.330 


10 


1.04 




6ns 




















34 


34 


3.40 


CM 


Navy ns 

6.000 StMlMU^ 

16.000 SMMUt 


2.300 
2.300 


14.3 
14.3 


4.020t 
6.fiJ0 


34 

22 


0.07 

0.04 1 



•2.8ai hm 9« Mhi p« mrlw 8 yun. ^'^S-^i 

'C^ilrii Prti C«»tnHH< pmaw, fcf yriimi riMBMimm 14 iU|wt 1171. 
'aiMi M 7n acivt iMiM cMtraM. 

IralMli ai M itiMt p» ivaiMl. 1177. 
•320 MhH M so ttiMs m tnM, 1177. 
3'2ft*7»-3ft 



System hardware costs can be expressed In three ways: (1) 
system procurement cost, (2) cost per terminal connected, and 
(3) cost per student-hour (over some chosen amortization period). 
In terms of system procurement cost, a range between near $35,000 
(the stand-alone GETS) and over $10 million (a 1,000-termlnal 
PLATO IV system) can be noted, a factor of close to 300 times. 
On a per-termlnal basis, though, available Information Indicates 
an Inverse relationship between system size and cost. As an 
example, for the TICCIT system, the per-termlnal system cost of 
the 32-termlnal configuration Is close to two-and-one-half times 
that of the 128-termlnal configuration. This Information Indi- 
cates a substantial economy of scale for larger systems. 
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A more meaningful relationship in comparing computer-based 
instruction with other methods is the cost per student-hour. 
This cost is inversely proportional to terminal utilization rateo, 
to system life span and, for computer-managed instruction, to the 
number of students a terminal can accommodate. Realized costs 
per student-hour will be highly sensitive to each of these. An 
average of 2,000 hours per year per terminal is a widely cited 
target value, but one which appears difficult to attain. Shoul.; 
it prove attainable, and assuming a system life spsn of 5 years. 
Indicated student-hour costs for CAI systems range between 
roughly $1.00 (the 128-terminal TICCIT system) and $3.50 
(GETS). The lo;*er per student-hour ■ cost associated with large 
systems implies a large initial commitment of funds (if central 
hardware is purchased) and a large commitment to CAI with the 
other costs and risks it entails. Assuming that each CMI termi- 
nal would accommodate 50 students, student-hour costs would 
appear to be less than $0.10. 

Note that the $3-50 associated with the GETS is based on 
information that is several years old. Systems of comparable 
capability, incorporating recent technological advances iyi micro- 
processors and data storage devices, can be anticipated to coso 
considerably less. 

9- Program Delivery; Communications 

Communications are relevant only for large systems where 
terminals may be geographically separated from central pro- 
cessors. In current military applications, these are limited 
to PLATO IV and the Navy CMI system. Communications have been 
accomplished through two modes— microwave transmission and land- 
lines, but microwave transmission has received too little dis- 
cussion to allow characterization of its costs here. The rate 
schedule for communications over Commercial long lines, as 
reported In two studies, is shown in Table 16. 
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TABLE 16. RATE SCHEDULE FOR COMMUNICATIONS OVER 
COMMERCIAL LONG LINES 



DISTANCE 
BiTERVAL 

(Sf^ES) 


COST PER INCREMENTAL 
MLE PER MONTH 


AVERA6E COST PFR MILE 
PER MONTH (AT UMIT OF 
DISTANCE INTERVAL 


1-25 


$3.30 


$3.30 


26-100 


2.31 


2.56 


101-250 


1.65 


2.01 


251-500 


1.15 


1.58 


>50Q 


0.83 


1.20« 



•At tOCa mttt. 

M fi?^ JinysM (1973), imI MAMm. Papttti, and licM (1974). 



For land-lines, a conunonly used rule of thumb is $1.00 per 
mile per month ^or long lines (interstate), but line distances 
of greater than 1,000 miles are required before costs decrease 
to this level. Rates charged government agencies for lines 
leasv i through the General Services Administration are typically 
stated at half the commercial rate, and $0.50 per mile per month 
ts the ^aluG commonly used in military studies. The significance 
of communications costs for a large system with a central com- 
puter, can be appreciated by the following. At the $0.50 per 
mile per month rate, estimated communications costs in the PLATO 

V experimental programs averaged over 50 percent of computer 
rent-.l and i arminal maintenance costs. 

10. Program Delivery; Materials 

No data were found on these costs of instruction. 
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n . Program Delivery; Facilities 

During time periods in which the military services are not 
expanding, the need for additional facilities and furnishings 
should be a minor consideration in instructional costs. Cases 
where such costs would be incurred would be limited to the intro- 
duction of new training courses (as might accompany the intro- 
duction of new operational equipments) and major changes in the 
hdy Instructional material is presented. In both cases, re- 
quirements might be levied for modifying and outfitting instruc- 
tional areas (classrooms and laboratories) with fixtures to 
accommodate new training equipments. 

This appears to be the case in transitions from conventiona." 
to either individualized or computer-based instruction, but such 
costs appear to be modest (see Table 17). The introduction of 
individualized or computer-based instruction would normally re- 
quire replacement of traditional classroom desks with carrels 
and itiight require the extension of electric service to individual 
student positions and the conversion of classrooms to larger 
learning centers. Introduction of computer-based instruction 

TABLE 17. COSTS OF FACILITIES: LEARNING CARRELS, 
ELECTRIC AND PNEUMATIC LINES 



ITEM 


COST PER UNIT 


REFERENCE 


Individual Learning Carrel 






30 Carruls 


$90 


Caiman, DeLeo, «f4^ (1977) 


20 Carrels 


260 


Steinkerchner, Deignan, atd (1977) 


Eleciric and Pneumatic 






Lines (PLATO IV) 






30 Carrels 


$61 


Dallman, Deleo, at aL (1977) 


20 Carrels 


141 


Steinkerchner, Deignan, atd (1977) 
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might require other utility service; for example, PLATO IV termi- 
nals require compressed air and communication lines. 

12 • Program Management and Administration 

No data were found on these costs of instruction. 

i 3 . Student Personnel; Pay and Allowances and Other Costs 

Data on pay and allowances are widely available. Neverthe- 
less, such data appeared only in three studies (see Table 13). 

An alternative to training at schools is to conduct the 
same training at operational sites. It has been proposed that 
computer-based instruction would increase the amount of opera- 
tional site training that is feasible and avoid costs of relo- 
cating personnel to the schools. Relocation costs were treated 
by only one study. Polcyn, Baudhuin, et at., (1977) present 
data that permit estimation of transfer costs (including per 
diem) per course: $i»25 for advanced training based on permanent 
change of station, $400 foj.- advanced training based on temporary 
duty transfer, and $l40 for initial training based on permanant 
change of station. A significantly higher cost ($825) for perma- 
nent change of station for advanced training is cited by the Air 
Force in "USAP Cost and Planning Factors" (Air Force Regulation 
173-10) 

C. DISCUSSION AND SUMMARY 

There is no evidence that one method of instruction is most 
cost-ef fec»/ive for all types of military training. The most 
cost-effective method for a particular situation will depend 
upon such factors as type of course material, location of instruc- 
tion, numbers of students, and life-span of the training. It is 
apparent that the cost data currently available make x impossible 
to examine satisfactorily the conditions which would make a par- 
ticular method of instruction the most cost-effective alterna- 
tive . 
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Unless there are key studies that have been overlooked, both the 
quantity and quality of current data on training costs are meager. 

The training-management and data-reporting systems currently 
employed by the Services do not provide Information needed for 
evaluating the cost-effectiveness of alternative methods of in- 
struction for two reasons. First, they provide Information only 
to the level of complete training courses, while analyses of the 
cost of methods of Instruction need data that can be associated 
either exclusively or predominantly with a single method of in- 
struction within a course. Second, training courses use many 
resources provided by school management and organizations at 
higher echelons (i.e., "training support"). The use of such 
resources may differ significantly between different methods of 
instruction. However, current reporting systems veil the cost 
differences by allocating support on arbitrary bases, such as 
averaging across all students located at a training facility. 

Data reporting systems that would provide information suit- 
able for cost-effectiveness analyses of instructional methods 
would be more complex and expensive than current systems for two 
reasons. The first is the straightforward multiplication of the 
number of training activities whose costs may be separately 
identified and compiled; that is, each course has many segments. 
The second reason lies in the structure of military training. 
If the costs of training support functions are to be attributed 
In other than an arb4.trary manner, they must be initially re- 
corded in a way that empirically associates the support provided 
with the individual course segments receiving the support. Con- 
sidering the size and complexity of military training organiza- 
tions, this is a task of great magnitude. 

Two steps are necessary to lay a foundation for building a 
data base on the costs of training. The first is to formulate 
hypotheses regarding causal relationships between training pro- 
gram characteristics and resource requirements in order to 
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identify the types of data that should be collected. The second 
step is to formulate and evaluate alternative schemes for col- 
lecting the data. 

Table l8 identifies, for hypothetical purposes only, some 
determinants of costs for various methods of instruction, based 
on our review of the literature, analogies to weapon system 
costs, and intuition. The determinants shown are respresentative, 
rather than exhaustive, and probably encompass only the more 
obvious factors. At that, the table shows the extensive range 
of cost and non-cost data required to assess training costs in 
a manner suitable for analyses of the cost-effectiveness of 
methods of instruction. 

We note that training cost data could be collected in three 
fundamentally different ways: universally (continually on all 
training activities), by a sampling procedure, or on an ad hoc 
basis. We have discussed the nature of each of these methods 
but have not evaluated them or the costs and benefits associated 
with each alternative. The question of how- best to collect data 
on the costs of alternative methods of instruction is a central 
issue requiring further and thorough study. 



80 

93 



TABLE 18. HYPOTHESIZED DETERMINANTS OF COSTS 



BCSmCE TYPE 01 RMCTVW 


AmJCABU 
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HYWTWKED KTBOMUUCn OF COSTS 
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IMmn 
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V. DISCUSSION 



Computer-assisted and computer-managed instruction for 
military training have been evaluated in about 30 studies con- 
ducted since 1968.. Most of these were experiments of limited 
duration. In a few cases, CAI still remains in use, e.g., 
PLATO IV for training medical technicians at Sheppard APB, Texas, 
and vehicle repair mechanics at Chanute APB, Illinois, and 
TICCIT for training S-3A tactical coordinators at Naval Air 
Station, North Island, San Diego, California. There have been 
fewer evaluations of CMI systems but most of these systems are 
still operating after ^ or more years, e.g., the Navy CMI, Air 
Force AIS, and Army CTb. A wide variety of courses, involving 
both the acquisition of knowledge and performance skills, were 
included in these evaluations. 

A. MAJOR FINDINGS 

The principal findings are summarized in Tabic 19. Computer- 
assisted and computer-managed instruction are as effective as 
conventional instruction when measured by student achievement 
at school, but a more direct and relevant measure of effective- 
ness is the performance of graduates on ,1obs in operational 
units. Correlations between performance in school and on the 
Job, though thought to be high, have not been demonstrated either 
for computer-based or conventional instruction. 

Computer-based instruction typically saves 30 percent or 
more of the time students need to complete the same courses given 
by conventional instruction. The amounts of time saved range 
widely, but research has not addressed the factors that could 
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TABLE 19. SUMMARY OF FINDINGS ON CAI AND CMI, COMPARED 
TO CONVENTIONAL INSTRUCTION 



Measure 


FMng 

(CotTipared to Conventional Instniction) 


Comments 


CAI 


CMI 


Student AcMevement 


Same or mere 


Same 


Performance measured only at schooL 
Relation between per! urmance at school 

and on the fobnot demonstrated. 
Observed dHfererices not of practical 

importance. 


Coim CompietkNi 
Time 


Nad 

Comperisons 


40 


8 


CIMI: Most time savings mair^tained 
or increased witii extended use. 


Time saved 
(Median) 


29% 


44% 


Range 


31 to 89% 


12 toG9% 


No. of 
Comparisons 


5 


7 


■ 

Cc^^niter support saves littie time beyond 
that of mdhnduaized instruction. 


Ttme saved 
Infividuat 
ized In- 
struction 

CAI 


64% 

69% 


51% 
CMI 51% 


Student Attrition 


About the same 


Sight increase 
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account for these variations. Thus, no explanation can be given 
at present for the different amounts of time savings found between 
CAI and CMI, between various types of CAI, between various types 
of CMI, between different courses, between different types of 
Instructional strategies (e.g., drill and practice, tutorial, 
simulation, student pacing algorithms, types of remediation) and 
the like. 

It Is widely believed that transforming a course from con- 
ventional to Individualized (or self-paced) Instruction saves 
student time. Three explanations are generally offered for this 
effect: 

• Paster students are not held back by rates of presenting 
material In conventional Instruction set to permit 85 to 
90 percent of the students to complete the course. 

• Course materials are reviewed and Irrelevant materials 
tend to be eliminated when courses are modified In format 
from conventional to Individualized Instruction. 

• Special remedial materials can be provided to students' 
on the basis of Information gained by frequent diagnostic 
testing of their progress through a structure of rela- 
tively brief lessons. 

Computer-assisted and computer-managed Instruction are, of 
course, forms of Individualized Instruction. Little attention 
has been given to the Incremental benefits. If any, that computer 
support may bring to Individualized Instruction (without computer 
support). Some data were found where student performance could 
be compared on the same courses given by conventional. Individ- 
ualized, and CAI or CMI Instruction. Student achievement at 
school was about the same with each method of Instruction. 
Individualized versions of five courses saved 6^1 percent of the 
time required by conventional instruction; the CAI version saved 
an additional 5 percent or a total of 69 percent. For seven 
other courses, the individualized and CMI versions each saved 51 
percent of the time required by conventional instruction. 
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Transforming a course from conventional to Individualized 
Instruction Is expected to save student time, as occurred here. 
But, It Is also clear that the addition of computer support to 
the Individualized versions of these courses does not further 
Increase to any appreciable degree the amount of student time 
saved. The extent to which shortening the course may have con- 
tributed to saving student time cannot be determined from these 
studies. Since the same Instructional material, both In content 
and In structure, was provided In the Individualized and com- 
puter-based yerslons of these courses, there Is no special reason 
to expect tha' the addition of computer support should produce 
any Incremental time savings. However, there Is a substantive 
question as to whether the Incremental cost of computer support 
In these cases produced Incremental benefits. The particular 
studies from which these data were taken did not address this 
Issue. It Is not Implied here that computer support per se does 
not produce benefits equal to or greater than Its cost. Computer 
support to an Instructional program may bring certain unique 
benefits such as reducing the number of Instructors and support 
personnel needed for Instruction, reducing the costs of main- 
taining student records, and reducing the costs of modifying 
and updating courses because of an ability to keep detailed 
records on student performance. Whether the costs of adding 
computer support (CAI or CMI) to Individualized Instruction 
(without computer support) produces benefits equal to or greater 
than these costs Is an Issue that clearly needs careful explora- 
tion. 

B. COST-EFFECTIVENESS OF CAI AND CMI 

There have been only a few attempts to^ estimate the cost- 
effectiveness of CAI and CMI and these are based on Incomplete 
analyses of the costs of Instruction. Table 20 summarizes the 
results of these studies. All of them are based on the premise 
that the amount of student training time saved by a method of 
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TABLE 20. SUMMARY OF STUDIES REPORTING COST SAVINGS AND/OR COST- 
EFFECTIVENESS OF VARIOUS METHODS OF INSTRUCTION 



MBiim 01 

btttrucbun 


SystHn 


Samicc 


Locition 


;m 


Nontef of 
Coum 


Nimbor of 

StudMrtsin 
ExponHMRts 


Nonbor of 
SludHrts AsMRid 
fof C&tiiHiti 


EstiniiMl 
Sawngs 
Pv Ymt 


Rofwtnco 




PUTOIV 


A 


AMm 




3 


536 




PIATO IV Mt CKMffKlNt^ 


OS Afwr Oirinwi C». «i 

ScMOITS) 


CM 


PUTOIV 


N 




17 


1 


22 


2B0 pkHB p« y«r 


I0.57M* 


Crawfta^ HviKh* Plifli 
«iSHnMOI70) 




PUTOIV 


AF 




19 27 


• 


1211 


37Spir«Nk 


PIATO IV MM H OKt^ffKliM 


OriteM, TMtt. HUi Mii 


(rmnd c«ni) 




SO 


4 


410 


300 ptr cUu 


% or 


CmWi GralMi, HvA% 
ttiinOTS} 




Nwf CUM 


N 




41-78 


4 


400 


300 ptr class 
ptr wttk 


$ 3Nr 


uwi^ oraMii^ mii^ 




Nwy CMI 


N 










SZ472 gretem' 

(KM 


9UMFY75' 
IIMFYTI' 
IIOJOMFYT?' 


BiMmi MiwU niTV 
Brirtil MMrUtll/D 
Bririii«MM(imi 


CMH 


MS 


AF 




24 35 


4 




2M2t 

(KM 


14171^ 
$IM(4vn) 


M.1. 1f74«i^1. 1171 
Bridii«MlvU(imi 




MS 


AF 


Uwry 


ias2 


4 




5511 


710 


OcLl. 1in-Sirt3n 1170 




MS 


AF 


Uwry 


•If 1L5* 


4 






AIS OMftHKitfi tif iMii 
li inMnKtv-Hppvtirf wV- 
pKii« M mm cant, m( ii 
tdNn! ccnpvlv cmii miI in 
CNfiriMi li Mlar KM CM 


llTtJily 1S70 
MSS««ietT«t 
Brtifii«Mlmri (11701 



* Ih» to canrnMRiGstion and MRlMm cam; PUTO N 

- I • '1' i n iilttiiiftpiriilwiliininnh Hw <Mii i ii m iiiiiM w cMiatta. 

'Pwnwi fraw cat w >oid wM al IS.7W owr 10 yi i ■ m iMi^H rt i o aiw y nip h aii ow iym i to pwida friHiwa 

tiandPUTOIVtBn>MB;dNS3Ac»^MriagravMaBlv0paMitaftlM Boaahiawaawarttaoit 

woofhii^ r d J ty ii Ma te iBfdialBliyatadCaBrt S Y i^ 
' B a caw of yao ia r dawle y w u ai md apai a lhy u a t a tar PUTO IV. 
*Caniparad to convontnnal iMvclian bafore faviiion. 

2.7.7I.4 



Mfs^ aomy 10 oaoraa iwnwt 

'Coaiiiuidaiiuia 



*Can|MriaaB flf oMMfliy aaV^ocod nairaciian CMI M 
**Darivad by piMtjag otiiRMta rinvm aboM. 



4-17-7I 



99 



ERIC 



Instruction provides major cost savings; the amounts of cost 
savings are estimated by computing the pay and allowances of 
students for the amounts of student time saved In training; the 
resultant amounts should more properly be called "cost avoidance 
savings". This procedure was applied to time savings due to 
PLATO IV, Navy CMI, and AIS and. In one case, to revised course 
materials In a course given by conventional Instruction. Pour 
of tnese studies consider other costs In addition to those 
avolc^ed by student time savings, such as for preparing course 
materials, purchase or use of computers, and the number of In- 
structors required by each method of Instruction (Crawford, 
Hurlock et al. y 1976; U.S. Army Ordnance Center and School, 1975; 
Dallman, DeLeo et at., 1977; and the AIS Service Test, described 
In an Air Force briefing, 1978). 

The dollar amounts of such "savings" could be large, depend- 
ing, of course, on the number of students assumed for these es- 
timates, e.g., about $10 million a year for about 50,000 students 
Instructed In FY 1977 by the Navy CMI system and about $3 million 
a year for about 5500 students Instructed In FY 1978 by the Air 
Force AIS system. According to two cost-effectiveness evalua- 
tions that have been reported, the PLATO IV system Is judged to 
be not as cost-effective as Individualized Instruction. These 
conclusions are based on Incomplete cost data In two small-scale 
tests (535 students In four courses at U.S. Army Ordnance Center 
and School, Aberdeen Proving Ground, Maryland, 1975; 1261 students 
In four courses at Chanute AFB, Illinois; Dallman, DeLeo et at. y 

1977) . The Air Force AIS was found to be cost-effective, compared 
to Instructor supported, self-paced Instruction In one course 
(Inventory Management) but not In three others; the computer costs 
which made the latter courses not cost-effective were judged to 

be small In comparison to other school costs (AIS Service Test, 

1978) . Since all of these findings are based on Incomplete cost 
data, the findings cannot be generalized or even taken seriously. 
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other benefits, beyond those of saving & training 
time, are often said to occur with CAI and CMI, . ^ely because 
the computer can compile records and direct the ai." :jntion of 
instructors, on the basis of various algorithms. The following 
list is illustrative rather than complete: 

• More precise data for improving and upd^^.'.ng course 
materials 

• Improved control over equipment, facilities, and materials 
for instruction 

• Improved allocation of resources among students 

• Improved ability to accommodate fluctuations in student 
loads 

• Increased student : instructor ratios, as well as the 
ability to use some instructors with less advanced quali- 
fications 

• Reduced need for support by noninstructional personnel 

• Reduced time of students on base waiting for courses to 
start 

• Reduced time of students on base waiting for orders after 
completing courses 

• Improved integration of records of 'students at school 
with those in central, computer-based personnel files 

• Improved utilization of instructors. 

Many of these benefits may occur with the use of CAI and 
CMI. None of them have been included in any cost-effectiveness 
evaluation known to us. Records kept at Lowry APB for students 
instructed by the AIS show that, compared to prior periods, they 
spend less time waiting to enter a course and waiting for an 
assignment after completing a course. Records kept by the Navy 
CMI system show that the average on-board count of students in 
school has been reduced for those instructed by that system: 
the extent to which this may be attributed to various benefits 
has not been examined. 
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C, WHAT SHOULD BE DONE NEXT? 

The potential value of computer-assisted and computer- 
managed Instruction for military training rests primarily on 
findings that (1) computer-assisted and computer-managed Instruc- 
tion save 30 percent or more of the time (median value) required 
by students under conventional Instruction and that (2) student 
achievement at school Is about the same with computer-assisted 
and computer-managed Instruction as with conventional Instruc- 
tion, However, these results do not necessarily Imply that 
computer-assisted and computer-managed Instruction are cost- 
effective because of fundamental problems with the measures of 
effectiveness and of cost used In the studies from which these 
results are taken. Effectiveness, as measured by student achieve- 
ment at school. Is not necessarily a measure of performance by 
course graduates In relevant jobs after they leave school. Data 
on the costs of alternative methods of Instruction reported In 
various studies are essentially Incomplete, particularly with 
respect to courseware, student : Instructor ratios, support and 
management services; this applies both to computer-based and 
conventional Instruction. The results that have been reported 
are limited to obvious costs observed during experiments (e.g. ^ 
preparation of courseware, rental of computers) and do not con- 
sider long-term costs associated with operational applications 
(e.g. ^ numbers of Instructors and support personnel, revisions 
to course materials, maintenance of software and facilities, 
management). Next, we discuss steps that should be taken to 
remedy these deficiencies. 

1. Measures of Effectiveness 

There Is a need to compare performance on the Job of stu- 
dents Instructed In the same courses by alternative methods of 
Instruction. In practice, comparisons will be required between 
conventional, lndlvl(;iuallzed, and computer-assisted or computer- 
managed Instruction. The general absence of objective data on 
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the performance of students on Jobs after graduation from mili- 
tary training courses Is a major deficiency of research on many 
aspects of military training and Is not limited solely to deter- 
mining the cost-ef xectlveness of computer-based Instruction. 
(See McCluskey, Trepagnler, Cleary et at., 1975; Pickering and 
Anderson, 1976; and Foley, 197^, 1975, for recent efforts on 
measurement of Job performance In the Army, Navy, and Air Force, 
respectively. Note: "... major assessment programs, either 
within or outside the military, that rely on performance tests 
as their primary data source are almost non-existent." Pickering 
and Anderson, 1976, p. 3*) It Is also Important to collect on- 
the-job performance data for several time Intervals after stu- 
dents leave school (e.g., 3, 6, and 12 months) In order to observe 
the short as well as longer-time effects of different methods of 
Instruction. It may turn out that time saved at school must be 
compensated by spending more time In training on the Job and that 
deficiencies In performance on the Job attributed to one method 
of Instruction disappear relatively quickly. Thus, there may 
well be a variety of trade-offs between the costs and benefits 
of various methods of Instruction and amounts of training In 
schools and on the Job. 

It would be a major undertaking to develop objective methods 
of measuring performance on the Job and to collect on-the-job 
performance data. If the school-Job correlations are found to 
be high, we would have a basis for accepting student achievement 
at school as a proxy for the measurement of performance on the 
Job. At present, we do not know the extent to which sUch cor- 
relations may exist and. If they do, that they have about the 
same magnitude for various methods of Instruction, for various 
types of courses, and for varying periods of time on the Job 
after leaving school. There should also be a feasibility study 
to examine the advantages, disadvantages, and costs of various 
methods of collecting and reporting on-the-job performance data. 
Although these data are proposed here to evaluate the 
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effectiveness of various methods of training, they would also be 
useful for other purposes such as (1) setting standards for 
recruitment and advancement and (2) estimating the technical and 
maintenance readiness of the operational forces. 
2. Measures of Cost 

Available cost data are unsatisfactory for conducting cost- 
effectiveness evaluations of computer-based and other methods of 
Instruction used In military training. Data on the costs of 
Instruction provided by recent analytical studies of military 
training are Incomplete. Further, since such cost data come from 
experiments that were limited with respect to amounts of course- 
ware, numbers of students, and duration of the experiments. It 
Is questionable whether the results should be used to' estimate 
the costs of In&tructlon under operational conditions. Data 
collected through the Services' current cost-reporting systems 
are not satisfactory because costs are Identified only with 
complete training courses and the costs of training support are 
allocated to courses on arbitrary bases not related to actual 
utilization or requirements by specific courses. Cost element 
structures for collecting such data are Identified In this paper 
and elsewhere. (See Petruschell and Carpenter, 1972; Braby, 
Henry, Parrlsh, and Swope, 1975; Seldel and Wagner, 1977; and 
McDonnell Douglas Astronautics Company East, 1977c). 

A comprehensive effort to collect data on all the relevant 
costs on all methods of Instruction used In military training 
would be a very large effort, m effect, however. Identification 
of the major cost drivers for various methods of Instruction Is a 
necessary condition for assessing the cost-effectiveness of 
feasible alternatives. Decisions about using or not using new 
methods of Instruction must be made now without benefit of 
reliable cost data. However desirable, a large-scale effort to 
collect such data cannot be recommended now without further 
examination of Its scope, benefits, and costs. Instead, Initial 
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efforts should be directed towards evaluating alternative ways 
of developing a data base. In addition, cost data should be 
collected on major Instructional systems that have recently come 
Into use and on new ones being considered for procurement. These 
systems are Identified below: 

a. Air Force Advanced Instructional System (AIS) , The 
current AIS Incorporates capabilities for research that would 
not be needed In an operational version. Thus, cost data are 
needed on an operational AIS and on alternative methods of In- 
struction for technical training. The recent AIS Service Test 
(unpublished as of February 1979) suggested that AIS was cost- 
effective compared to Instructor-managed Instruction only In 
one of four courses used In that evaluation. 

b. Navy Computer Managed Instruction System (Navy CMI) . 
Published Information suggests that the Navy CMI system saves 
student time and thereby avoids costs. However, complete cost 
data on this system have not been published. For cost- 
effectiveness evaluation, cost data are also needed for compar- 
able courses using Individualized and conventional Instruction. 

c. Navy Aviation Training Support System (ATSS) . This 
planned system will support computer-managed Instruction for 
enlisted men and officers at 20 Naval and Marine Corps Air Sta- 
tions; additional units, not yet planned, could support Naval 
surface warfare facilities. Cost data will be needed to support 
cost-effectiveness evaluations of this system and the methods 

of Instruction that It would replace. 

'd. Army Automated Instructional Management System (AIMS) . 
The Army AIMS Is based on the Navy ATSS and the above remarks 
about cost data also apply here. About 20 units will be ac- 
quired If the Initial Instv llatlon at the Field Artillery School, 
Fort Sill, Oklahoma, Is found to be effective. 

e. Marine Corps Communication-Electronics School CAI/CMI 
System . The Marine Corps plans to procure a CAI/CMI system for 
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the Marine Corps Communication-Electronics School, Twenty-nine 
Palms, California. The initial configuration of this system 
calls for installing 26o terminals in 3 years with a potential 
growth to manage 2000 students and a maximum of 1000 terminals. 
A preliminary cost analysis is being conducted by the Navy Per- 
sonnel Research and Development Center. 

3. Research and Devel opine nt 

An "Integrated DoD Plan for R&D on Computers in Education 
and Training" was prepared in draft form by a tri-Service group 
in September 1975. This plan should be revised and brought up 
to date on the basis of more recent information now available 
on computer-based instruction. Next, consider steps that should 
be taken to improve our ability to provide more relevant and 
accurate information on the effectiveness and cost of various 
aspects of computer-based instruction. 

3« Factors Which Influence the Amount of Student Time 
S^ved. The data show that computer-assisted and computer-managed 
instruction save appreciable amounts of student training time 
compared to conventional instruction. However, there is great 
variation in the amounts of student time savings found in many 
studies; extreme values of -31 to 89 percent have been reported. 
Other things being equal, the military Services should obviously 
favor those applications of computer-based instruction which 
promise greater student time savings. Thus, research is clearly 
needed to explore the conditions which influence the amount of 
student time saved. Factors which could influence the amount of 
student time saved by CAI and CMI probably include the quality 
of the course materials (for which metrics should be developed 
and standards set), types of courses (i.e., some may save more 
time than others), and instructional strategy (e.g., effects of 
combinations of drill an-^ practice, simulation as a method of 
instruction, frequency of testing, length and difficulty of les- 
sons, and methods of managing students' rates of progress through 
a course). On a longer-term basis, it is important to know 
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whether 'itudent time savings accomplished at school bring any 
penalties, such as In qusdlty of performance on the Job and a 
need for additional on-the-job training. 

b. Student Attrition with CMI , Currently available data 
suggest that student attrition with CMI may be somewhat larger 
than that with conventional Instruction. However, It Is not 
clear that the observed Increases In attrition are due primarily 
to computer-managed Instruction because the qualifications of 
students In these courses appeared to drop at the same time. 
Other factors may also be Involved, e.g.y changes In the number, 
quality, and support provided by Instructors. Put simply, there 
Is a need to determine the extent to which CMI and other factors 
may Increase attrition of students, compared to the rates that 
occur with other methods of Instruction. 

c . Role of Instructors in Computer-Based Instruction . The 
role of Instructors probably differs significantly in convent o 
tlonal, individualized, computer-assisted, and computer-managed 
Instruction. Yet the benefits to be. derived from each method of 
instruction surely requires that instructors perform adequately 
the particular functions required of them in each case. Only a 
few studies consider the attitudes of instructors to CAI or CMI; 
all of these are unfavorable in comparison to conventional in- 
struction. The limited amount of data cannot be regarded as 
conclusive. Thus, there is a need to develop more reliable in- 
formation on the attitudes of instructors to all methods of 
instruction used at present by the military Services. The survey 
instruments should be diagnostic in nature so that steps could 

be taken later to remedy problems that may be identified, e.g.^ 
experience and training of instructors, relevance of this train- 
ing to their Jobs as instructors, and areas where instructors 
believe that problems exist. Emphasis should be given to 
(1) computer-managed instruction, since that is in greater use 
by the military Services than is computer-assisted instruction 
and to (2) individualized instruction, because this method of 
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Instruction is a reasonable alternative to conventional as well 
as to computer-managed Instruction. On a longer-term basis, 
steps should be taken to identify what instructors should do to 
make computer-assisted and computer-managed instruction most 
effective at least cost . 

^' P.omparison of Individ ualized and Computer-Based Instruc - 
iion. A significant finding of this paper is that, compared to 
conventional instruction, computer-based instruction saves no 
more time than does individualized instruction. Student achieve- 
ment in school was about the same in all cases. The obvious 
question is whether computer-assisted and computer-managed in- 
struction provide benefits, greater than those of individualized 
instruction, that are worth their incremental costs. On the 
surface, it does not appear that the additional savings in stu- 
dent time obtained with computer support would be sufficient to 
pay for the incremental costs. However, this observation does 
not consider significant cost savings that computer-based in- 
struction might bring in a reduced need for instructors, improved 
record keeping and management of students, instructional materials, 
and the like. An analysis of the costs of individualized, 
computer-assisted and computer-managed instruction for the same 
courses is very desirablp. 

®- Methods of Collecting Cost Data on Methods of In<;truc - 
tiojl- There will be a continuing need to collect various types 
of cost data on alternative methods of instruction. It is clear 
that current management and reporting systems do not provide data 
that are satisfactory for use in analyses of methods of instruc- 
tion. Three methods of collecting cost data appear to be avail- 
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• Universal collection 

• Sample collection 

• Ad hoc collection. 
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A feasibility study Is needed to examine the advantages, 
disadvantages, and costs of various ways of collecting cost data 
that will be needed to support cost-effectiveness evaluations of 
methods of Instruction In the near future. 

Other Types of Research and Development . Certain types 
of research and development might Improve the effectiveness and/ 
or reduce the costs of computer-assisted and computer-managed 
Instruction In military training but there Is an Insufficient 
basis, at present, to recommend funds for their support. Major 
Items of this type would probably Include the following: 

• Further development of "Intelligent computer-assisted 
Instruction". This refers to the use of the computer 
to model each student *s style of learning and to use 
this Information to construct lessons best suited to 
his unique needs from detailed materials stored In the 
computer. In effect, this eliminates the need to pre- 
pare complete lessons for storage In the computer, as 

In present CAI systems; It may also Improve the effective- 
ness of CAI In Instructing Individual students. 

• Improved methods of preparing courseware to reduce high 
costs currently encountered In this area. 

• Improved video discs and solid-state memories to signifi- 
cantly reduce the costs of major components In computer- 
assisted and computer-managed Instruction, particularly 
for stand-alone Instructional systems used away from 
schools . 

• Improved communications for computer-assisted and 
computer -man aged Instructional systems which use a large 
central computer to support many terminals at different 
locations . 

• Investigate the feasibility of developing a general model 
for use In cost-effectiveness studies of military training. 

• Determine the maximum acceptable costs of stand-alone 
terminals or other system-design concepts for them to 
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be cost-effective In selected applications of CAI and 
CMI, i.e., to establish cost goals for the Improvement 
of technologies useful for computer-based Instruction, 
such as video discs and solid-state memories. 
• Define ways of measuring various benefits claimed for 
CAI and CMI. It has been suggested that CAI and CMI 
provide various benefits not now available with conven- 
tional Instruction. These Include, for example. Improved 
control of materials and facilities required for Instruc- 
tion, Improved utilization and assignment of Instructors, 
more accurate Information derived from computer records 
to Improve lessons and tests, more accurate and complete 
student records, and the like. No data have . been offered 
to support such claims and their Impacts are not clear 
on the costs or the effectiveness of computer-based In- 
struction. An exploratory study would be useful to 
define ways of measuring various potential benefits of 
computer-based Instruction and of Identifying the cost 
and other data that would be needed to- estimate the 
magnitude of such benefits. 

Given the absence of precise Information on the major 
cost-drivers In computer-assisted and computer-managed Instruc- 
tion, for either large-scale or small-scale Installations, It Is 
difficult to establish any amounts or priority for funding 
research and development on most of the Items noted above. Nor 
Is It clear how much Improvement Is needed or Is feasible In any 
of these areas In order to make a significant Impact on the cost 
and/or effectiveness of computer-based Instructional systems 
that would Incorporate such Improvements. It may also be noted 
that some of these studies, no matter how desirable, cannot be 
undertaken until more detailed cost data become available. 
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METHODS OF INSTRUCTION 



There are many methods of instruction, such as lecture, dis- 
cussion, tutoring, independent study, and the like; 20 different 
methods of teaching listed in the Rand Corporation's "Method of 
Designing Instructional Alternatives (MODIA)" (Carpenter-Huffman 
1977). More than one method of instruction may be used in any 
course. For purposes of this study, we group methods of instruc- 
tion into four general categories, as follows: 

Category Examples 

Conventional instruction Lecture, discussion, 

demons trati on 

Individualized instruction Programmed instruction, self- 
paced instruction, preci- 
sion teaching 

Computer-managed instruc- Advanced Instructional Sys- 

tion (CMI) tem (AIS) 

Navy Computer Managed In- 
struction System (Navy CMI) 

Computer-assisted instruc- PLATO IV, TICCIT, GETS, LTS 
tion (CAI) 

These categories embody several key distinctions: conven- 
tional and individualized instruction do not require computer 
support; CAI and CMI do, and are generically referred to as com- 
puter-based instruction (CBI). Conventional instruction is 
aimed (by definition) at instructing groups of students at the 
same pace (counselling and tutoring of individual students are, 
of course, attempts to individualize instruction within the 
structure of conventional instruction); the other three methods 
are designed to permit each student to learn at his own pace. 
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Each of these methods of Instruction is described below. 

A.l CONVENTIONAL INSTRUCTION 

Conventional Instruction typically consists of lectures and 
discussion In which all students are supposed to learn the same 
material at the same rate; it is sometimes referred to as "lock- 
step" Instruction, platform Instruction, group scheduling, or 
block scheduling. A shortcoming of this method is Its relative 
Inflexibility, particularly with large groups of students. 
Students differ In their rate of learning and they enter a 
course with varying degrees of knowledge about the material to 
be taught. However, Information Is presented at a constant 
rate for all. Even If the rate Is one at which most members 
of the class can learn, it will necessarily penalize those for 
whom that rate Is either too fast or too slow. Thus, slow 
learners may progressively fall further behind and perhaps fall 
and fast learners may simply lose Interest In coming to class. 
All students spend about the same amount of time In the class 
and at the completion of the course differences between them 
are reflected In how much has been learned, -i.e.. In final 
grades from norm-referenced tests. 

A. 2 INDIVIDUALIZED INSTRUCTION 

Individualized Instruction Is a way of arranging a curric- 
ulum Into small lessons and tests so that each student can pro- 
ceed at his own rate. Although the term "Individualized In- 
struction" could also apply to computer-based instruction. Its 
use here will be limited strictly to various forms of Individual- 
ized Instruction conducted without a computer. There are many 
types of Individualized Instruction that do not rely on com- 
puters such as self-paced instruction, programmed Instruction, 
personalized system of Instruction, and precision teaching; 
these differ primarily in the Instructional strategies and In 
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the role assigned to the instructor in order to provide each 
student with the type of materials best suited to his style of 
learning. 

,The term "self-paced" needs to be qualified. Obviously, 
no student is permitted an unlimited amount of time to finish 
a course; students are encouraged to request assistance from 
the instructor when some difficulty is encountered or the in- 
structor may intervene when he observes that a student is pro- 
ceeding slowly. Various incentive schemes may be employed, 
such as assigning a completion date to each student, the post- 
ing of average class progress, or assistance to slower students 
by faster ones. Other differences among methods of individual- 
ized instruction concern whether all students must progress 
through the same set of lessons ("straight-line") or whether al 
ternative lesson materials are provided ("branching"). 

Lessons can be presented in booklets, by audio-visual de- 
vices, in a laboratory set-up, or in work situations such as a 
maintenance shop; the lesson material can consist of knowledge 
or skill or both, as in the maintenance, calibration, and re- 
pair of equipment. Instruction is oriented to the complet^e 
mastery of lessons; lessons and tests are tied to each other. 
If a student passes a test, he goes on to the next lesson; if 
not, he repeats the lesson in the same or modified form. The 
student can take tests without taking lessons since successful 
completion of a test is presumed to show that he knows the re- 
quired material; he need take only the lessons prescribed for 
the tests that he failed. It is not a trivial matter to deter- 
mine the proper amount of information to be included in a sin- 
gle lesson, to decide whether a slide or a written text is the 
more effective way of presenting certain information, and to 
arrange a sequence of lessons that is efficient for instruction 
al purposes. The essence of individualized instruction is to 
provide a structure of lessons and tests that a student can 




take at his own rate to master an entire course. It does not 
require the use of a computer. 

To graduate, all students must pass the required lesson 
and end-of-course tests. The Instructional material Is pre- 
pared so that there Is a high probability that all students 
can complete the course. Students either know what they are 
supposed to know or they do not graduate, as determined by cri- 
terion-referenced rather than norm- referenced tests. Differ- 
ences among students are reflected primarily In the amount of 
time they need to complete the course, although grades and test 
scores may also be recorded. 

A. 3 COMPUTER-MANAGED INSTRUCTION (CMI) 

Individualized Instruction permits each student to proceed 
at his own pace, but It generally Increases the Instructor's 
clerical and advisory loads. The Instructor needs to give and 
score more tests than he would In a conventional classroom. 
He must keep track of each student's rate of progress on almost 
a dally basis and work closely with students when they have 
problems with any part of the course. He must find the right 
lesson, audio-visual cassette, or test bench whenever a student 
needs It and keep track of all available resources. All of 
these functions, and certainly all of the clerical and bookkeep- 
ing tasks, can be accomplished readily by a modern computer (see 
Baker 1978 for a recent review). 

In computer-managed Instruction (CMI), all Instruction 
takes place off-line, i.e.^ away from the computer. The point 
of contact between the student and the computer Is the test 
which accompanies each lesson. The test may be given either on- 
line or off-line with a machlne-scorable answer sheet. In 
either case, the computer scores the test. Immediately reports 
the results to the student, tells him which lesson to take next 
and where It may be found In the learning center. The next 
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lesson, of course, may represent .progress or remediation; if 
there is too much of the latter, the computer advises the stu- 
dent to consult the instructor. 

A CMI facility can provide many of the services associated 
with training such as the following : 

• Testing 

-Scoring 
-Diagnosis 

-Prescription, such as drill and practice, or remedia- 
tion (repetition of all or part of a lesson, or 

assignment of the student to parallel tracks that 

differ in the level of difficulty between st^ps or 

in the method of presenting information). 

• Management 

-Monitoring 

-Pacing students according to predetermined rules 

• Scheduling 

-Assignment of students 

-Optimum allocation of instructional materials and 

facilities 
-Assignment of instructors 

• Administrative record keeping 

-Student personnel records 
-Student test data, graduation 
-Instructor records 

-Inventory and control of learning resource materials 

(films, tapes, workbooks, projectors, etc.) 
-Frequency and time of use of materials and facilities 

• Course development 

-Since test results show how students answered each 
test item, information is available to identify 



the difficulty of the different parts of each lesson. 
This provides a basis for modifying the course and 
testing the effectiveness of various improvements. 

Examples of CMI systems are: 

Army ; Automated Instructional Management System (AIMS) 

Field Artillery School, Port Sill, Oklahoma 

Navy : ^ Computer Managed Instructional System 

Naval Air Technical Training Center, 
Mlllington, Tennessee . 

Aviation Training Support System (ATSS) 
[planned for about 20 Naval and Marine Corps 
Air Stations; previously called Versatile 
Training System (VTS)]. 

Air Force : Advanced Instructional System (AIS) 
Air Force Technical Training Center 
Lowry Air Force Base, Denver, Colorado 

A. 4 COMPUTER-ASSISTED INSTRUCTION (CAI) 

In Computer-Assisted Instruction (CAI), the instructional- 
materials are stored in and presented by the computer. Each 
student interacts individually with segments of the material 
and takes tests through some type of terminal. The typical ter- 
minal contains one or more output and input devices; for out- 
put, a television or plasma panel display, screen for slides or 
fiche, paper printer, loudspeaker; for input, a keyboard, tele- 
typewriter, pointing-type capability, graphic tablet, microphone. 

CAI systems generally include many terminals. In the TICCIT 
(Time-shared, Interactive, Computer-Controlled, Information Tele- 
vision) system, up to 128 terminals and one control computer are 
located"^ at the same site; in the PLATO IV system (Programmed 
Logio for Automatic Teaching Operation), about 1000 terminals 
in different locations are linked to a central computer by long- 
distance communication lines. In principle, each CAI terminal 
could have its own mini- or micro-computer, but such an arrrange- 
ment is expensive at present. Systems of this type (called 
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"stand-alone") have been developed for the military Services, 
e.g.j the Lincoln Terminal System (LTS) and the General Electric 
Training System (GETS). 

A narrow definition of CAI would consider it to be a way 
("medium") of presenting instruction, comparable to TV, slide, 
or filmstrip and regard its flexibility as a form of CMI • How- 
ever, this distinction has little practical meaning. The inter- 
active nature and flexibility of a CAI system distinguish it 
from CMI or programmed instruction of any variety. With appro- 
priate programming, CAI permits a dialogue to occur between 
student and computer on every frame of an instructional se- 
quence. The computer can tutor, prompt, drill, and test the 
student on a frame-by-frame basis; by means of simulation, it 
can guide and test the student on complex dynamic processes. 
By diagnosing the student's progress, it can identify and select 
the material best needed to meet specific deficiencies; it could 
compose (in fact create) appropriate lessons and tests from a 
large store of elemental materials. 

Although CAI is defined narrowly as a medium of presenta- 
tion, most CAI systems also provide the administrative and re- 
cord-keeping capabilities inherent in CMI systems. 

A developing application of computers to instruction is 
"Intelligent Computer-Assisted Instruction" (ICAI). In ICAI, 
subject matter knowledge is stored in the computer, but not in 
the form of previously defined lessons. Instead, the computer 
models each student and selects stored subject matter material 
to construct instructional interactions for individual students. 
These Interactions can be very flexible, and hold some promise 
for "human-like" tutoring of students. 

ICAI is a very new area with work currently underway, spon- 
sored by DARPA, ONR, and others. However, no data are available 
yet on costs or effectiveness, and therefore, ICAI will not be 
treated further In this paper. There can be little question 
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about the superior instructional capabilities provided by a 
CAI system. The real issue is whether these capabilities are 
worth their cost. / 



Examples of some CAI systems evaluated in military training 
are listed below. These are described in Appendix B. 



CTS 



PLATO IV 



TICCIT 



LTS 



GETS 



Computerized Training System 
U.S. Army Signal School 
Port Gordon, Georgia 

Programmed Logic for Automatic Teaching 
Operation 

Computer-based Education Research Laboratory 
University of Illinois 

Time-shared, Interactive, Computer-Controlled, 

Information Television 
The MITRE Corporation 

Lincoln Terminal System 
Lincoln Laboratory 

Massachusetts Institute of Technology 
Lexington, Massachusetts 

General Electric Training System 
General Electric Ordnance Systems 
Electronics Systems Division 
Pittsfield, Massachusetts 



The idea of individualized instruction, which underlies 
all forms of computer-based instruction, long precedes the dev- 
elopment of modern computers. It was promoted by John Dewey 
about 1900 at the University of Chicago (and by Socrates cen- 
turies ago). Sidney L. Pressey (1926, 1927) designed several 
mechanical "teaching machines"--which provided preprogrammed 
drill and practice frames as well as automatic self-scoring of 
tests. His concept was to provide immediate feedback and self- 
pacing in education. Testing one of these machines. Little 
(193^) found that 
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students immediately apprised of their test 
results, and given opportunity to correct deficiencies 
by make-up tests, profit markedly in terms of final 
advaJitage°" students who do not have such 

Students. . .given opportunity to correct de- 
ficiencies by drill and by make-up tests, likewise 
so profit. 

The greatest benefit accrues to students who usually 
score in the lower half of the distribution, although 
the entire group moved upward. 

(Reprinted in Lumsdaine and 
Glaser, 196O, p. 65) 

Pressey expected his machines to produce a "coming indus- 
trial revolution in education" but this did not occur. In 1932, 
he wrote, "The writer has found from bitter experience that one' 
person alone can accomplish relatively little and he is regret- 
fully dropping further work on these problems." (Quoted in 
Skinner, 1958). Skinner's teaching machines required the stu- 
dent to compose his response rather than, as did Pressey's, 
select it from a set of alternatives. Skinner presented mate- 
rial in a progression of small steps, each of which the student 
could probably understand and, in so doing, become rea''dy for the 
next; the student got feedback by being reinforced for every 
correct response. (Skinner, 195^, 1958). Skinner's work was 
done at the right time and had impact. By 1962, over 80 differ- 
ent teaching machines and 630 instructional programs were com- 
mercially available; six machines were computer controlled 
(Aeronutronics, DEC, Marquardt, Rheem, TRW, and USI Robodyne; 
Finn and Perrin, 1962). IBM simulated a Skinner teaching 
machine on an IBM 65O, starting in 1958, and work on PLATO 
started at the University of Illinois in 196O. (Rath, Anderson 
and Brainerd, 1959; Alpert and Bitzor, 1970). Some noted engi- 
neers contributed to this progress. In a well-known article, 
Vannevar Bush (19^5) predicted a computerized desk which would 
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contain a large volume of library materials and monitor a 
reader's progress. In 1957, Simon Ramo described how "teaching 
engineers" with pushbutton classes and memorizing machines could 
help meet the Increasing need for more education In a growing 
technical society. The Department of Defense (AFOSR, AFPTRC, 
NTDC, ONR) supported much of the original R&D on teaching 
machines during the 1950s. 

A. 5 ADVANTAGES AND DISADVANTAGES OF VARIOUS METHODS 
OF INSTRUCTION 

The Services conduct Individual training at 126 different 
facilities (76 for specialized skill training alone); each 
Service offers 3000 to ^000 courses. The average number of 
students on board In these courses can vary at any time from 
about 10 to 2,500 (the largest Is for a course In propulsion 
engineering at the Navy Training Center, Great Lakes, Illinois). 
Course length can vary from days to months. It Is not likely 
that any method of Instruction Is best suited for all courses 
In this wide spectrum of requirements. With this obvious quali- 
fication In mind. It Is useful to compare the advantages and 
disadvantages generally attributed to the four methods of In- 
struction that have been described. The main points are summa- 
rized In Table A-1. 

A. 5.1 Conventional Instruction 

Conventional Instruction permits flexibility In presenta- 
tion of material to suit the needs of Individual students, pro- 
vided the Instructor Is free to do so. Human contact can serve 
to motivate students. The standard rate of progress Is estab- 
lished to produce some goal established by policy, e.g.^ that 
at least 90 percent of the students master the course; the slow 
learners may fall too far behind to catch up; the fast learners 
waste time and may lose Interest. Individualized attention 
becomes Increasingly difficult as class size Increases. 



TABLE A-1. ADVANTAGES AND DISADVANTAGES OF FOUR 
METHODS OF INSTRUCTION 



METHOD OF 
INSTRUCT I DN 



ADVANTAGES 



DISADVANTAGES 



Conventional 
Instnjction 



Individualized 
Instruction 



Lo*» delivery cost for large class sizes. 

Flexible In mode of Instruction, use of media, course 
content, emphasis. 

Direct human contact. 

Simplifies planning because all students must pro- 
gress at the same rate and complete the course at 
the same time. 

Instructors prepare Instructional material. 

Grades at end of course define how much each student 
has learned in relation to a normative sample. 



Explicit course and lesson objectives. 
Standardized Instruction. 

All students progress at their own rate {i.e.^ slow 
learners do not hold up faster learners). 

Students can skip course material they already know, 
as shown by preassessment tests. 

Testing and evaluation closely tied to small lesson- 
steps. 

Lessons generally one track. 

All graduates are warranted to know the required 
Information {i.e., students pass the required 
tests or they do not graduate). 

Instructors can concentrate their time on those 
students who need assistance at both ends of the 
distribution. 

Permits use of Instructor's aides, thereby reducing 
average level of qualification required of Instruc- 
tors. 

Permits wide use of different Instructional media. 

Instructors relieved from rote repetition of basic 
materials. 

Instructors can have time to address concepts as 
well as student evaluation, motivation, and 
enrichment. 



Not all students are able to progress at same rate 
with equal mastery. 

Low student: Instructor ratios Increases costs. 

Individual attention to students difficult as class 
size Increases. 

Differences b?t>«en Instructors may lead to non- 
uniform achievement. 

Fast learners may lose Interest In course. 

Slow learners become increasingly penalized. 

Load on Instructor for scoring tests and managing 

students' progress increases markedly with class 

size. 

Difficult to Insure student mastery of training ob- 
jectives since norm-referenced rather than objective- 
referenced testing procedures are used. 

Difficult to Insure that Instructors present the 
relevant Instructional Information. 

High Initial costs for development of course materials, 
carrels, audio-visual equipment, etc. 

Increases demand for qualified personnel to prepare 
Instructional materials. 

Requires changes in the Instructor's role in conven- 
tional Instruction. 

Load on Instructor for scoring tests and managing 
students' progress Increases markedly with class 
size. 



CMI 



All of those for Individualized Instruction, plus: 

Reduces demand for number of Instructors. 

Presentation of lessons and taking of tests not 
dependent on computer. 

Automated test scoring, evaluation, prescription. 

Student progress monitoring. 

Multi-track lessons readily handled. 

Automated student management, record-keeping 
and scheduling. 

Resource management. 

Detailed Information routinely available for 
evaluating and modifying lessons and tests. 

Manual scoring possible If computer and/or 
connunlcatlon falls. 

Predict gradua^'lon date, based on rate of student 
progress in course and personal data In 
student's file. 

Provides data base for research, course develop- 
fnent, and management decisions. 



All of those for Individualized Instruction, plus: 

High Initial costs for courseware, CPU, terminals. 

High operating costs for communications, where 
required. 

Instructional material poorly matched to students' 
abilities and expectations may discourage students 
and reduce effectiveness. 

Instructor's attitudes often unfavorable. 

Scoring and student management Inoperative If com- 
puter and/or communications fall. 



(continued ) 
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TABLE A-1 . (Continued) 



CW A wide array of courses can be offered with few 

Instructors (or, at remote facilities, without 
Instructors, limited primarily by cost factors 
and availability of communications). 



ADVAWTAGES DISADVANTAGES 



All of those for CMI — except that ability to 
operate when computer Is Inoperative Is extremely 
limited — plus: 

Very flexible means for presenting material and 
talcing tests via con^uter. 

Interactive tutorial modes feasible. 

Simulation of processes and equipment feasible. 

Computational aids readily available. 

Can provide detailed Information needed to im- 
prove specific lessons and tests, e.g., 
student success with various subjects, method 
of presentation (graphic, text), instructional 
strategy, delay times. 

Can provide Instructors with data bases, formats, 
guidelines for developing Improved course ma- 
terials. 



All of those for CMI , plus: 

Instruction becomes difficult when computer re- 
sponses are delayed. 

No instruction possible when computer and/or 
cofmunications fail. 



Facilitates maintaining security of tests. 

Probably the greatest degree of individualized 
instruction currently available, except where 
very low studentrinstructor ratios are accept- 
able. 
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Instructors favor this perhaps because the role offers high 
visibility and a position of authority. Low investment costs 
are required for current courses because, generally speaking, 
they were paid long ago when courses were originally developed. 

A teacher can reach a large audience in a lecture hall and 
multiply this reach almost without limit by means of a movie or 
TV recording. However, as McKeachie (1970, p. 13) said in a 
frequently quoted comment: 

The technological bottleneck in education 
is that we have no device that allows a teacher 
to listen or respond to more than one student 
at a time. 

A . 5 . 2 Individualized Instruction 

Individualized instruction, whether delivered by programmed 
texts, a multi-media approach, or by computer, has the advantage 
of dealing more efficiently with different rates of learning and 
different amounts of prior knowledge among students. Its ma.lor 
limitations lie in the efforts required to 

• Identify the specific lesson ob.lectives which, in 
som- progression, satisfy the overall course 
ob.1ectives 

• Develop instructional material that delivers the 
required information 

• Develop tests that measure the student's progress on 
each lesson and diagnose the types of remediation 
that may be required, and 

• Prepare the remedial treatments. 

Course materials must be pretested with students to ensure that 
lessons are neither too easy nor too difficult for students; it 
is often necessary to modify lessons in order to "validate" 
them. Developing courses for individualized instruction requires 
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qualified subject matter experts and curriculum development 
experts who are not always available when needed. 

Individualized instruction inherently makes more efficient 
use of the student's time than does conventional classroom in- 
struction since each student can start a course when he arrives 
at a school without having to wait for a class of some optimum 
size to be formed. Each student can graduate when he has 
mastered the specified material. Because each lesson has a 
test, the instructor can identify students who are falling be- 
hind early in the course, glvi them personal guidance, and pro- 
vide them with material selected to deal with their particular 
problems. This contrasts with conventional Instruction where 
tests tend to measure student competence but do not provide a 
basis for remediation; tests are also given less frequently. 

In this environment, an instructor must grade tests more 
often than in a conventional classroom; this in itself can be- 
come a large load. Since the instructor has more detailed 
knowledge about each student's progress, he can provide more 
individualized guidance than is possible in a conventional 
setting. 

A. 5.3 Computer-Managed Instruction 

CMI provides a means to handle many of the administrative 
loads encountered in individualized instruction, such as scoring 
and prescribing lessons, identifying students who neeci remedia- 
tion, managing and scheduling Jns tructional resources, and pre- 
dicting course completion ti^es that students can be sent 
promptly to their next asjs i.gnment . CMI systems readily comple- 
ment and can be tied into automated manpower and personnel 
management systems that are used by all military Services. 

A < 5 . 4 Computer-Assisted Instruction 

All of this can also be provided by CAI . The preparation 
of course material for presentation by CAI is similar in concept 
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but often more sophisticated than that for programmed instruc- 
tion or for CMI. It is obvious that it is probably inefficient 
to use computers to present programmed materials {i.e., to "turn 
pages") unless, of course, it costs less to do so by computer 
than by printing the same material on paper (which may soon be 
the case if frequent reprinting is required to update instruc- 
tional material and if the costs of word-processing types of 
computer systems continue to be reduced). The major advantage 
of CAI is that it permits extraordinary flexibility in querying 
and prompting each student, a process which permits the computer 
to select material of a complexity or level of difficulty most 
likely to meet each student's rate of learning and best suited 
to deal with his mlsund . landings and errors. Such a dialogue 
is highly motivating a. --rves to engage the attention of the 
student. It is indeed possible that a student may get more 
individual attention from a computer than he may, get from many 
human instructors, particularly where large classes are involved. 
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TYPES OF COMPUTER-ASSISTED AND COMPUTER-MANAGED 

INSTRUCTION SYSTEMS i 

B.l ARMY COMPUTERIZED TRAINING SYSTEM 
B.2 PLATO IV 
B.3 TICCIT 

B.4 AIR FORCE ADVANCED INSTRUCTIONAL SYSTEM 

B.5 NAVY COMPUTER MANAGED INSTTUJCT>0N 

B.6 STAND-ALONE SYSTEMS 

B.6.1 Lincoln Terminal System 

B.6. 2 General Electric Training System 
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TYPES OF COMPUTER-ASSISTED AND COMPUTER-MANAGED 
INSTRUCTION SYSTEMS 

Almost every existing computer-based instructional system, 
whether CAT or CMI, is tailored to some particular requirement 
such as the number and types of courses, location of school(s), 
student flow, and the availability of instructors, resources, 
and funds. Hardly less important is the influence of such fac- 
tors as whether the method of instruction should be primarily 
CMI or CAT, and whether the required computer, wherever it may 
be located, will be used solely for instructional purposes or 
also for some noninstructional purposes, such as maintaining 
medical and personnel records, base accounting, preparing pay- 
rolls, and the like. For such reasons, there are no "standard" 
computer-based instructional systems and few are likely to be 
entical. 



An inventory of computer-based instructional systems in 
current use by the military services does not exist. In a 197i| 
survey. It was found that computer-based instruction was used by 
the Army in 21? courses, the Navy and Marine Corps in 102, and 
the Air Force in 210. Compared to all other military instruc- 
tion, this accounted for 2, 3.6, and 2 percent, respectively, 
of all courses (Sherron, 1976). In another survey of ll6 Army 
courses at 16 schools, also in 197^, use of the computer in these 
courses varied over the widest possible range, i.e., from 0.01 
to 100 percent (Rich and VanPelt, 197^). 
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The military Services have conducted experiments on most 
of the CAI and CMI systems that have been available since about 
1965* Even the most recent acquisitions, such as the Air Force 
Advanced Instructional System or the Navy Computer Managed In- 
struction system, represent computer technology of the early 
1970s. Only the major features of the systems used In military 
studies are described here (see Sherron 1975 for additional 
Information) . 

B.l ARMY COMPUTERIZED TRAINING SYSTEM 

The Army Installed a prototype Computerized Training System 
(CTS) at the U.S. Army Signal School, Fort Gordon, Georgia, over 
the period 197^-1976. The program to evaluate CTS for Army use 
was called Project Abacus, a name used Interchangeably with CTS. 
The Army refers to CTS as a CAI/CMI Instructional system. There 
are 128 terminals In CTS, each with a Visual Display Unit and a 
keyboard. CTS also contains six mini-computers (PDP-11/35); 
four of these computers, called Display Controllers, support 32 
CAI terminals each; the two other computers serve as System 
Controller and Data Base Controller, respectively. 

The CTS features a fast response time: each of the 32 
terminals In a cluster can be updated In less than 250 milli- 
seconds. Three courses were developed for CTS: 

• Field Repair Radio Course (31E20) 

• Teletypewriter Equipment Repair Course (31J20) 

• Avionics Communications Equipment Repair Course (35L20). 

CTS was applied to these courses at Fort Gordon after 
feasibility and follow-up studies (conducted at the U.S. Army 
Signal Center and School, Fort Monmouth, New Jersey, during ' 
1968-1971) showed that CAI Is as effective or better than con- 
ventional Instruction for training In basic electronics (Longo, 
1972). Those Initial tests were conducted with the IBM I500 
Instructional System, using the IBM Coursewrlter II language- 
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Results of the Fort Monmouth tests are described elsewhere in 
this report; results of the Port Gordon tests were published 
after this report was completed (see Seidel, Rosenblatt, V/agner, 
Schulz and Hunter, I978). 

B.2 PLATO IV 

Development of the PLATO system (Programmed Logic ^for Auto- 
matic Teaching Operations) began in I96O under the leadership of 
Donald Bitzer at the Computer-based .Education Research Labora- 
tory, University of Illinois (called Coordinated Science Labora- 
tory until 1967). (See Computer-based Education Research Lab- 
oratory, 1977; Smith_and Sherwood, 1976; Lyman, 1977). PLATO 
IV, the current version, uses a large central computer (CDC 
CYBER 74) at CERL which supports 950 terminals at about 150 
locations throughout the United States and one in Sweden. Other 
PLATO systems are located at Control Data Corporation, Arden 
Hills, Minnesota, and at Florida State University, Tallahassee, 
Florida. The basic architecture of the PLATO IV system can 
support up to 1008 terminals; a Computer Interface Unit controls 
data communication between the central computer and up to 32 
site controllers, each of which can support up to 32 terminals 
via direct connection or telephone line. The PLATO terminal 
contains a touch-sensitive display panel, keyboard, and micro- 
fiche projector; it can also control various multi-media devices 
that are attached to it. PLATO was developed with the support 
of the National Science Foundation, the Office of Naval Research, 
the Defense Advanced Research Projects Agency, the National 
Institute of Education, and Control Data Corporation. Control 
Data Corporation now offers CDC PLATO and PLATO Author Language 
on a commercial basis; these are production versions of the 
PLATO system and TUTOR language developed at CERL. 
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The PLATO IV system Is the most highly sophisticated, inter- 
active, Individualized Instructional system currently available 
(excluding various experimental devices still under development). 
It provides tutorial Inquiry, drill and practice, dialogue modes 
of Instruction, dynamic . simulation, and many types of computa- 
tional services and games. The TUTOR programming language con- 
tains over 250 commands which fall Into five large groups: dis- 
play, calculation, branching, answer Judging, and data collecting. 
A wide variety of data on student performance with various seg- 
ments of curricula and tests are available to Instructor and 
management personnel for analytical and management purposes; the 
system supports the development of Instructional material. The 
current CDC catalogue lists over 8OO courses and games that are 
available on a commercial basis. 

The military Services have evaluated PLATO IV In studies 
conducted at the U.S. Army Ordnance Center and School, -berdeen 
Proving Ground; Navy Recruit Training Command, Chanute Air Force 
Base; and other locations. These are summarized elsewhere In 
this report. The National Science Foundation supported a demon- 
stration program with PLATO IV at five community colleges In the 
area of Chicago; a total of 11 courses In selected areas of 
business, biology, chemistry, English, and mathematics were de- 
veloped for this program (Murphy and Appel, I977). In 1976, 
about 80 organizations (12 military) had dedicated communications 
lines to PLATO (CERL); PLATO (CDC) serves many universities and 
commercial organizations as well as Its own learning centers. 
(See Computer-based Education Research Laboratory, 1977, p. 37.) 

In addition to Instruction, the PLATO IV system presently 
provides a broad set of services, such as: 

• Electronic mall. 

• • On-line communications. Including text, graphics, 

and animation. 
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• Entertainment, Including games, musical presentations 
and simulation. * 

• Personal services. Including medical, financial, 
psychological, educational and career planning. 

• Research computation. 

• On-line research. Physical experiments are con- 
trolled by the PLATO terminal, and analyzed results 
are displayed graphically In real time. In addition, 
educational and social research can be conducted on- 
line and In real time. 

• Data processing. 

• Information retrieval. 



B.3 TICCIT 

The TICCIT system (Time-shared, Interactive, Computer- 
Controlled, Information Television) was developed, starting In 
1971, by the MITRE Corporation, with support from the National 
Science Foundation. C. Victor Bunderson at the Institute for 
Computer Uses In Education, Brlgham Young University (previously 
at the University of Texas), and M. David Merrill, at Western 
Montana College (previously at Brlgham Young University) were 
closely associated with this development, primarily using courses 
In freshman- level mathematics and English. Hazeltlne Corporation 
has offered TICCIT on a commercial basis since I976. 

TICCIT was designed to provide complete courses of Individ- 
ualized Instruction via computer on a lower cost basis than 
appeared possible with existing PLATO and IBM systems. Wherever 
possible, TICCIT used commercially available rather than specially 
designed components. The basic system uses two mini-computers to 
support up to 128 terminals and maintain records for up to 3000 
students. One computer serves as a main processor, the other as 
a terminal processor (both are Data General Nova 8OO). The ter- 
minals consist of a color TV receiver, teletypewriter keyboard, 
function keys, and a light pen; graphic and audio-visual material 
can also be presented. As In all CAI and CMI systems, the student 
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controls the pace of instruction. TICCIT courseware is struc- 
tured on a modular basis; within each segment of instruction, 
the student can select material to be presented at different 
levels of complexity and in formats that provide either the 
basic rule, examples, or practice. The TICCIT system provides 
authors with one Instructional strategy (that of learner control) 
in order to simplify the task of programming; this differs from 
PLATO where TUTOR offers several types of instructional pro- 
cedures, e.g.^ inquiry, dialogue, and simulation. 

The National Science Foundation supported a demonstration 
program with TICCIT at two community colleges; Northern Virginia 
Community College (Alexandria Campus), and Phoenix College of 
the Maricopa County Community College District, Arizona; the 
courses selected for evaluation were mathematics and English 
(Alderman, 1978). Other TICCIT installations are at the Model 
Secondary School for the Deaf at Gallaudet College in Washington, 
D.C., and at Brigham Young University. In the Department of 
Defense, TICCIT has been used on an experimental basis to train 
tactical coordinators for anti-submarine warfare in the S-3A 
aircraft at the Naval Air Station, Nor h Island, San Diego; a 
mobile system was installed for evaluative purposes at the Air 
University, Maxwell Air Force Base, Mc-' gomery, Alabama. 

B.4 AIR FORCE ADVANCED INSTRUCT l t..AL SYSTEM 

Planning for the Advanced Instructional System (AIS) at 
Lowry Technical Training Center, Lowry Air Force Base, Denver, 
Colorado, started in February 1969, when the Air Force Human 
Resources Laboratory (Technical Training Division) published a 
plan for the development of a computer-managed, computer-assisted 
instructional system. The system was developed as a computer- 
managed instructional system and tested under a contract with 
the McDonnell Douglas Astronautics Company, May 1973 to De- 
cember 1977. The AIS was designed to be a prototype system; 
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it incorporates capabilities for research, development, test, 
and evaluation that might not be needed in an operational system, 
(McDonnell Douglas Corporation, 1977a, 1977b). 

Current (April 1978) capabilities of the AIS are as fol- 
lows : 



Training load 

No. of courses 
No. of students per day 

Hardware 

Computer 

Interactive terminals 
Management. terminals 
Student carrels' 
Media devices 

Media allocation 

Printed materials 

Audio/Visual Presentation/ 
Illustrated Text 

CAI (used for management, re- 
search and course develop- 
ment ) 



3000 

(^500 with additional 
terminals) 



CDC CYBER 73-16 
50 
11 
8^17 
500 



605? 
3S% 

2% 



AIS provide- s the following functions common to most CMI 
systems: 

• Printed feedback to students of total score on tests 
and of objectives failed on tests 

• Printed assignment to next lesson, including resources 
required 

• Learning center rosters and individual student progress 
reports 
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Resource management, including material in learning 
centers, use of carrels, audio-visual devices and remote 
terminals 

Advising instructors about students whose results on 
preassessment tests indicate potential problems 
Displaying or printing student course and preassess- 
ment records for counseling 

Providing course evaluation and test item evaluation 
summaries 

Student Progress Management (SPM): SPM predicts a tar- 
get completion time for each student for each block 
and for the entire course. The predictions are based 
on the student's aptitude, ability, and performance; 
students and instructors receive a daily feedback on 
each student's progress toward the target completion 
times. The purpose of SPM is to pace each student to 
work at a rate Judged to be within his capability. 
Individualized Instruction Assignment (IIA): IIA as- 
signs individual students to alternative modules of in- 
struction for a lesson in order to achieve maximum pro- 
gress by each student. An adaptive decision process 
considers the individual characteristics and past per- 
formance of each student (preassessment and within- 
course data), his current placement in the course hier- 
archy and the availability of instructional resources. 
Each student is assigned to those modules, among the 
available alternatives, which the algorithm predicts 
he will complete in the shortest time. Three methods 
of making this assignment were tested: a degression 
model, "learner's choice" and an heuristic method, 
(i.e., assignment based on logical rules). IIA is a 
capability unique to AIS at present. 
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Most of the effort required to develop AIS was needed to 
convert four courses from conventional instruction to self- 
paced form suitable for support by CMI. These four courses 
were selected to represent a cross section of all technical 
training and a wide range of student aptitudes and abilities 
in the Air Force. These courses account for about 25 percent 
of the total training load at Lowry Technical Training Center, 

Average course length and number of graduates in these 
courses for FY 1978 were: 



Courses on AIS 

Inventory Management (IM) 
Materiel Facilities (MF) 
Precision measuring equipment (PME) 
Weapons Mechanic (WM) 

Totals 



Average Number of Graduates 
Length FY 1978 



(weeks) (Projected) (Actual ) 

7 3000 2492 

6 900 743 

32 600 659 

13 3000 1514 

7500 5408 



B.5 NAVY COMPUTER MANAGED INSTRUCTION 

The development of the Navy's Computer Managed Instruction 
(CMI) system can be traced directly to work started in I967 by 
G. Douglas Mayo, then on the staff of the Chief of Naval Air 
Technical Training, Millington, Tennessee (Kerr, I978; Middleton, 
Papetti and Micheli, 197^). At that time, it appeared that com- 
puter-assisted instruction, such as provided by PLATO or the 
IBM 1500 Instructional System, would be effective in the sense 
of saving student time. However, it appeared that implementa- 
tion of CAI systems in the Navy would be too costly. Mayo's 
premise was that instruction in the Navy's technical training 
courses should be revised from conventional to individualized 
formats and that computers should be used to manage but not to 
deliver instruction. 
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The Navy began to Implement Its CMI system In 1973 and ex- 
pects to complete Its Installation by I98O. The system will 
handle 16,000 students In 24 separate schools at six Navy train- 
ing centers; there were 6,000 students In 11 schools at five 
training centers in 1978.* Each "learning center" (an area for 
about 100 students In a training center) has an optical test 
scanner (OPSCAN 17) and a General Electric Termlnet 1200 key- 
board/printer. Each school has access to a remote batch terminal, 
with high-speed printer and card reader, which serves various 
management functions, such as dally progress reports, class 
rosters, and the like. The schools are linked to a central pro- 
cessor (Honeywell Series 60, level 66 computer) located at the 
Management Information and Instructional Systems Activity, 
Mllllngton, Tenn. 

Based on actual student loads, cost-avoidance savings were 
estimated to be $9 million to $10 million per year from FY 1975 
to FY 1977; they are expected to continue at the latter rate 
when the system Is fully Implemented. The Initial savings result 
largely from reductions In student loads because of Improved 
management of student time by CMI; reductions In support billets 
are expected to occur In the future. Acquisition of the system 
will cost $23.5 million In automatic data processing equipment 
alone; the development of courseware represents an additional 
cost. Expansion of the system Is contingent on the rate at 
which courses can be Individualized. Instructional Program 
Development Centers have been established to develop and maintain 
these courses. 

In addition to the CMI system centered at Mllllngton, the 
Navy has CMI systems at the Naval Air Station, Lemoore, Cali- 
fornia (VA-122) and the Naval Air Station, Mlramar, California 
(VF-124); these are part of the Aviation Training Support System. 

^Based on data provided by Chief of Naval Technical Tralnlne 
(Code 0153), April I978. 
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The Automated Shipboard Information Management System is a test 
installation of a shipboard computer which also provides CMI 
services on the USS GRIDLEY (transferred from the USS DAHLGREN, 
NPRDC 1977). The Marine Corps had an installation at Twenty- 
Nine Palms, California that is inactive at present (I978) pend- 
ing receipt of a new computer. Use of a communications satellite 
has been considered to link the central CMI computer at Millington 
with ships at sea to provide CMI for training personnel away from 
schools (Polcyn, 1977). 

The Navy has also supported development of Conputer-Aided 
Instruction Study Management System (CAISMS) which uses PLATO 
IV to give reading assignments, to give tests on-line, and to 
maintain student records. These functions could also, of course, 
be performed less expensively by using mini-computers rather than 
PLATO. (Alessi, Anderson, Anderson et at., 197^, Nievergelt, 
Alessi, and Montague, I978) . 

More recently, CAISMS was evaluated in a Navy technical 
training course. A conventionally taught section in an interior 
communications course was augmented with CAISMS; this was in- 
tended to provide adjunct instructional activities, so that 
students could more appropriately manage their study. There is 
potential for considerable cost savings if courses are configured 
to take advantage of the flexibility offered by computer manage- 
ment (NPRDC Technical Report "A Computer-Based Study Management 
System: Implementation and Evaluation in a Navy Technical Train- 
ing School," in preparation). 

Also, CAISMS has been reprogrammed to run on a mini-computer, 
and in that configuration has managed the study of over HOOO 
students in a week of a Navy technical course. Reports describ- 
ing this implementation, and its cost, are being prepared at the 
time of this writing (December 1978). 
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B.6 STAND-ALONE SYSTEMS 

Every computer-based Instructional system needs terminals 
to deliver Instruction (as In CAI) or to score tests (as In CMI). 
Prom the student's point of view. It hardly matters whether the 
terminal Is supported by a computer that Is located In the termi- 
nal or elsewhere. By definition, a "stand-alone" Instructional 
system contains a terminal and computer In one unit which needs 
only external power to operate. Two stand-alone CAI systems 
that have been evaluated by the military Services are described 
here. They now are estimated to cost between $30,000 and $50,000 
per unit, without courseware. Interest In stand-alone CAI sys- 
tems will probably Increase when, as Is often predicted, the 
home entertainment market will make micro-computers, video 
storage discs, and solid-state memories available at lower cost. 

Stand-alone systems seem promising for training In loca- 
tions away from schools where there are relatively small student 
loads, few or no Instructors available on site, and a demand ex- 
ists for a large variety of courses. Stand-alone systems could 
provide and guide instruction and also provide administrative 
information, such as student progress and courses completed, in 
computer-compatible form, for communication to central personnel 
data files. 

B.6.1 Lincoln Terminal System 

The Lincoln Terminal System is a self-contained, interac- 
tive, computer-based training system developed by Lincoln Labo- 
ratory, MIT, Lexington, Massachusetts. The latest version, LTS-5 
(Butman, 1977) uses microfiche to store and project visual images 
in a conventional manner; the fiche can also store audio messages 
to accompany each visual frame (up to 750 microfiche, each with 
12 audio/visual pairs of frames per fiche). The user Interacts 
numbers and function keys. " A teletypewriter and touch panel 
could be added to the system but they are not in the present 
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version. The LTS processor supports several standard author 
programs, which may be either branching or nonbranchlng In form. 
The system Is designed particularly to teach facts, principles, 
and computational skills. 

Only a limited number of LTS terminals have been built 
(about HO). The system has been tested at Keesler Air Force 
Base, Blloxl, Mississippi (air traffic control operator course 
and an electronics principles course) and at Lincoln Laboratory 
(digital systems engineering; see Butman and Frlck, 1972; 
Butman, 1975; Butman and Kunze, I976). 

B.6.2 General Electric Training System 

The General Electric Training System (GETS) Is another self- 
contained, automated. Interactive, Instructional system (Rupp, 
1976; General Electric Ordnance Systems, 1976). The terminal' 
contains a plasma display panel, teletypewriter keyboard, func- 
tion keys, sonic pen, and a random-access. 35-inin slide projector 
(80 slides/tray). Floppy discs are used for lesson preparation 
and playback. The plasma screen and slides can be used for in- 
teractive training, e.g., using text, simulated control panels, 
or circuit diagrams. To date, GETS has been used for training' 
on operating and maintenance procedures for the TRIDENT weapon 
control system at the Guided Missile School, Dam Neck, Virginia 
and the Fleet Ballistic Missile Training Center, Charleston, 
South Carolina; it is scheduled for use at the Fleet Ballistic 
Missile Training Center, Bangor, Washington. There will be 
about 25 GETS units in the TRIDENT program. The TRIDENT program 
relies heavily on the use of tactical equipment and equipment 
simulation for training purposes. GETS will be used primarily 
to handle peak training loads that exceed the capacity of the 
available training equipment (called "laboratory" training in 
this program). GETS is viewed as cost-effective for procedural 
training in the TRIDENT prog.-am compared to the cost of acquir- 
ing additional tactical equipment. Current training loads on 
GETS are small, but larger loads are expected to occur in April 
1981. 

B-15 

■151 



APPENDIX C 

EFFECTIVENESS OF COMPUTER-ASSISTED I NSTRUCTION"" ' 
IN NONMILITARY TRAINING 

C.l. INTRODUCTION 

C.2. EFFECTIVENESS OF PLATO IV AT COMMUNITY COLLEGES 

C.3. EFFECTIVENESS OF TICCIT AT COMMUNITY COLLEGES 

C.4. ANALYSIS OF TIME SPENT BY STUDENTS ON TICCIT AND IN 
CONVENTIONAL INSTRUCTION AT COMMUNITY COLLEGES 

C.4.1 Student Contact Hours 

C.4.2 Student Activity Survey 

C.4.3 Calculated Total Times 



C-1 



15 



APPENDIX C 



EFFECTIVENESS OF COMPUTER-ASSISTED INSTRUCTION 
IN NONMJLITARY TRAINING 

C.l. INTRODUCTION 

This appendix considers briefly the effectiveness of CAI 
and CMI instruction in nonmil'tary settings, primarily in 
schools and colleges. Education in schools and colleges dif- 
fers from military training in one major characteristic of in- 
terest here: students in military training receive pay and al- 
lowances while they are being trained, while those in schools 
and colleges do not. This means that reducing the time spent 
at school could reduce the cost of military training while no 
such incentive exists in schools and colleges (at least at 
present). Other distinctions between schools and colleges and 
military training might also be drawn with respect to such 
factors as subject matter, tenure of instructors, and the rela- 
tionship of training to jobs and careers; however, these are 
not critical to the present discussion. 

The effectiveness of CAI and CMI in schools and colleges 
has been the subject of many excellent books and reviews such 
as the following: 

Kearsley (1975); Seidel and Rubin (1977); Salomon 
and Clark ,(1977); Davisson and Bonello (I976); Froomkin, 
Jamison, and Radver (1976); Levien (1972); Goldstein 
(197^); Edwards, Norton, et at. (197^); Jamison, Suppes 
and Wells (197^); and Baker (1978). 

Most of these are concerned with CAI which has received 
more attention in schools and colleges than has CMI. The issue 
of ei'fectiveness here is almost entirely on student achievement, 
that is, the amount of course materials acquired as measured by 
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tests; some attention has been given to the effect of CAI on 
academic attrition. 

The literature cited above suggests that CAI is an effec- 
tive supplement to regular instruction at the elementary school 
level; used as a replacement, it is about as effective as con- 
ventional instruction at the secondary school and college levels. 
The drill and practice and simulation modes of CAI are at least 
as effective as conventional instruction; some studies suggest 
that they are more effective than conventional instruction; 
the results are equivocal for the tutorial and problem-solving 
modes. The apparent differences in the effectiveness of various 
CAI modes may be the result of improper comparisons because, 
for example, CAI drill and practice is generally used to 
supplement , while the other modes are used to replace conven- 
tional instruction. Limited data suggest that CAI can reduce 
the time required for learning. According to Baker (1978), 
about 30 CAI systems are being used in academic environments at 
all levels of education. 

The National Science Foundation supported two large-scale, 
studies that evaluated the effectiveness of the PLATO IV and 
TICCIT systems in teaching basic courses at community colleges. 
Both of these CAI systems have been used experimentally by the 
military Services. These studies are summarized briefly here 
because of their iinportance and their potential relevance to 
the effectiveness of PLATO IV and TICCIT in military training. 
The studies do not consider the amount of time, if any, saved 
by students, the effectiveness of the particular coursewares 
in distinction to the delivery systems, or the cost-effective- 
ness of these two systems. A survey of student activities in 
the TICCIT study provides an indirect assessment of how much 
time students spent in conventional and CAI instruction. 
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C.2. EFFECTIVENESS OF PLATO IV AT COMMUNITY COLLEGES 

The effectiveness of the PLATO IV CAI system was evaluated 
at five community colleges In the Chicago area, 1972 to 1976, 
(Murphy and Appel 1977; Computer-based Education Research Labora 
tory, 1977). Most of the time during this period was required 
to develop course materials and tests and to conduct prellmln-- 
ary evaluations. The final evaluation was conducted during 
two semesters (1975-1976) In 162 classes In five subject-matter 
areas (accounting, biology, chemistry, English, and mathematics) 
at four of the colleges. Most comparable PLATO and non-PLATO 
courses were taught by the same Instructors, thereby holding 
constant the possible Influence of Instructors on student 
achievement, attrition, and attitudes. Instructors who parti- 
cipated In the test were not required to use PLATO IV for any 
specified amount of time or for any specified material; rather, 
they used PLATO IV In various ways to replace, supplement, or 
reinforce classroom Instruction. Average student use of 
PLATO IV varied from a few minutes to more than 20 hours for 
Individual students; In terms of courses, use of PLATO IV 
varied from 1 to 12 hours per course for 126 courses. The per- 
missive approach made PLATO IV very acceptable to the faculty 
but It complicates and makes It more difficult to extrapolate 
the findings of this evaluation to the more highly controlled 
environments of most military studies where students and In- 
structors had no option to shift back and forth between CAI 
and conventional Instruction. 

Bearing In mind the way In which the evaluation was per- 
formed, the following findings are significant: 

• Student achievement on PLATO IV was about the same as 
that for regular classroom Instruction 

• PLATO IV produced no notlcaable effect on student at- 
trition 

• Student and faculty attitudes to the use of PLATO IV 
were generally favorable 



• The development of curriculum materials was a diffi- 
cult undertaking. The test was postponed for one year 
while additional staff were added to the project to 
develop more PLATO IV lessons. 

C.3. EFFECTIVENESS OF TICCIT AT COMMUNITY COLLEGES 

The effectiveness of TICCIT was evaluated by comparing the 
performance of students In six mathematics and three English 
courses Instructed by TICCIT or conventional lecture, textbook 
and discussion (Alderman, 1978). The study was conducted In 
1975-1976 at Phoenix College, Arizona, and the Alexandria cam- 
pus of the Northern Virginia Community College. Two years were 
required to develop the course materials and achievement tests 
and to train the staff Involved In the evaluation. Over 260O 
students were enrolled In the TICCIT courses and 3000 In the 
lecture courses; In addition, about 300 students at Alexandria 
took programmed courses In mathematics without computer support. 
A measure of effectiveness which turned out to be important was 
the percentage of students who completed the course under each 
mode of instruction. 

Instruction by TICCIT was at least as effective as by 
lecture or by programmed material. Students Instructed by 
TICCIT had higher test scores (by about 10 percent) than those 
instructed by lecture In nine of twelve mathematics courses and 
in four of seven English courses; where TICCIT test scores were 
lower, the differences were quite small. 

Some of the differences that favor TICCIT may be attribut- 
ed to the related finding that students who completed the TICCIT 
classes were more highly qualified than those in the conventional 
classes. The much lower course completion rates on TICCIT, 
compared to the lecture course, are certainly related to the 
finding that only the more qualified students completed the 
TICCIT course. 



A surprise finding is the low completion rates for students 
taking courses on TICCIT, particularly mathematics. The over- 
all completion rates for mathematics were l6 percent for TICCIT 3 
50 percent for lecture, and 20 percent for programmed instruc- 
tion. When the data are adjusted to include students who did 
not complete the course or who withdrew and reenrolled, i.e.^ 
completed the course in other semesters), the adjusted comple- 
tion rates for mathematics on TICCIT are comparable to those 
for other methods of instruction. Completion rates were higher 
for English than for mathematics, but Instruction by TICCIT re- 
sulted again in lower completion rates than by lecture. 

This study shows that instruction in college algebra and 
English composition by TICCIT produces end-of-course results 
i.e. J test scores) that are equal to or higher than those as- 
sociated with conventional instruction. TICCIT appears more 
effective for algebra than English. These findings may be an 
artifact due to the dropout of the poorer students before the 
end of the course. TICCIT instruction appears more favorable 
for higher-aptitude than for average or lower-aptitude students; 
very few of the latter completed the courses on TICCIT in this 
study. The TICCIT study is one of the few that have examined 
CAI instruction in entire courses, under stabilized conditions, 
and on a large scale; its use in more than one location is also 
unusual. However, there is little reason to believe that the 
permissive atmosphere of a community college with respect to 
failure to complete courses provides a basis for comparison with 
military training. 

The amount of time required to complete courses by TICCIT 
or by conventional instruction, an important issue in military 
and industrial training, was not addressed directly in this 
study. However, the report contains survey data on how many 
hours students said they spent on course activities out of 
class. An analysis of these data suggests that students on 
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TICCIT completed these courses In less total time than did 
those in conventional courses. This analysis appears in the 
following section. 

C.4. ANALYSIS OF TIME SPENT BY STUDENTS ON TICCIT AND IN 
CONVENTIONAL INSTRUCTION AT COMMUNITY COLLEGES 

Two types of data reported by Alderman (1978) can be com- 
bined to estimate the total amounts of time spent by students 
to complete five courses instructed by TICCIT or conventional 
lecture and discussion. The data consist of (1) student con- 
tact hours (time spent on TICCIT or in class) and (2) a Student 
Activity Survey (in which students report how much time they 
spent on course-related activities in addition to the time in 
class). A description of these data and how they were used t*^ 
estimate total time spent by students on TICCIT and in conven- 
tional instruction follows. The results are shown in Table C-1. 

C.4.1 Student Contact Hours 

Data on student contact hours with TICCIT were compiled 
from records kept by the TICCIT system; data on hours spent in 
lectures are simply the result of scheduled hours for each 
class. Thus, for eight courses, a direct comparison can be 
made of the amount of time spent by students who used TICCIT 
and by students in class with conventional instruction. In 
five of the eight comparisons, students on TICCIT spent less 
time during the course than did those in class. These data 
are probably highly reliable but they do not include time spent 
on course work out of class. 

C.4.2 Student Activity Survey 

Data on student activities were collected only in /five 
courses. A survey of student activities included the following 
questions about time spent by students in each course: 
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TABLE C-1 . AMOUNT OF TIME NEEDED TO COMPLETE COURSES ON TICCIT 
AND CONVENTIONAL INSTRUCTION, 1976 
Source: Alderman "(1978) 



r 

1 










Student contact hours 


Student survey 


report, 


hours* 










Course 




TICCIT^ 


Lecture* 




TICCIT 


Lecture 


Calculated total 
times, hours 














Hours , 


























N 


Mean 


Hours 


N 


Hours 


N 


Hours 


TICCIT* 


Lecture* 


Hath 


007 


Beg. Algebra 


Phoenix 


Spring Semester 


(1) 

39 


(2) 
3S.6 


(3) 
37.5^ 


(4) 
20 


(5) 
97 


(6) 
120 


(7) 
116 


(8) 
48 


(9) 
107 




31 


Algebra I 


Alexandria 


Winter Quarter 


54 


35.9 


41. 7'* 


64 


87 


23 


123 


43 


114 




31 


Algebra I 


Alexandria 


Spring Semester 


27 


34.2 


41.7'* 


23 


91 


12 


101 


52 


96 


English 


19 


Basic Engl . 


Phoenix 


Spring Semester 


20 


30.6 


37.5' 














/ 


29 


Rev. Engl. 
Fundamentals 


Phoenix 


Spring Semester 


28 


35.1 


37.5' 
















19 


Basic Engl. 


Phoenix 


Fall Semester 


82 


38.9 


37.5' 


109^ 


141 


170^ 


102 




96 




29 


Rev. Engl . 


Phoenix 


Fall Semester 


40 


33.7 


37.5' 
















111 


Engl .Comp. 


Alexandria 


Fall Quarter 


66 


37.1 


25. 0^ 


115 


137 


74 


144 


110»^ 


105 



*Time spent on TICCIT system, students with post test, Table 7. 7, p. 34. 
•Classroom time, lecture and discussion. 
^Three meetings/wk x 15 wks x 5/6 hr/meeting. 
*Five meetings/wk x 10 wks x 5/6 hr/meeting. 
*Three meetings/wk x 10 wks x 5/6 hr/meeting 

♦Based on data in Appendix S, Student Activity Data. Total hours as reported by students for time spent on TICCIT or in class 
^tuL n^ r ^"^ "^^^^ ^"^^y ^^^^^^ ^or hours per week, multiplied by 15 weeks for semester 

(Math 007, English 19,29) or by 10 weeks for quarter (Math 31, English 111). All N's are approximate. 

'English ig and 29 combined. 

'Mean TICCIT hours (Table 7.7) plus student report for all non-TICCIT hours per week (Tables 7.1-7.4), multiplied by 
15 weeks for semesters or 10 weeks for quarters. 

•Total classroom time, as shown in this table, plus student report for all non-classroom hours per week (Tables 7.1-7.4). 
multiplied as above. 

^'Includes 18 hours attending regular class meetings without TICCIT. 

""Includes 31 hours attending regular class meetings without TICCIT. 
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"Approximately how many hours per week did you spend: 

(1) working on the TICCIT system/attending classes 
for this course? 

(2) in small group discussions about this course 
(outside of class)? 

(3) doing work for this course on your own away from 
TICCIT/working on homework assignments? 

(^) in total for this course?" 

The students' responses, in hours per week, were multi- 
plied by the number of weeks in the semester (15 weeks) or 
quarter (10 weeks) to get the total times shown in the table. 
In four of the five comparisons, students on TICCIT say they 
spent less total time to complete the course than did those in 
conventional lecture and discussion. Naturally, these compari- 
sons are based on the students' impressions and attitudes and 
cannot be verified. 

C.4.3 Calculated Total Times 

An estimate of the total time spent by students in each 
course was made by combining the documented time spent on 
TICCIT or in lecture (Columns 2 and 3) with students' reports 
of time spent out of class on course work (part of the total 
data shown in Columns 5 and 7, based on detailed data in the 
Appendix to Alderman's report). In four of the five compari- 
sons, students on TICCIT appear to have spent less total time 
in the course than did those in the conventional classes; 
the reverse effect occurs in one comparison. In three of the 
five cases, students on TICCIT appear to have spent about half 
the total time needed to complete the course that those in 
conventional classes did. 

Although there is no way to assess the reliability of these 
data on the times required to complete courses on TICCIT or in 
conventional classes, they are the only data we have. The ap- 
parent time savings on TICCIT would apply only to students who 
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completed the course (16 percent on TICCIT, 50 percent on lec- 
ture, as reported by Alderman). Those who completed courses 
on TICCIT had higher pre-test scores than their lecture counter- 
parts. Thus, the time and achievement advantages of TICCIT for 
the students described here do not apply to the majority of 
students who were unable to complete a course on the TICCIT 
program. 



/ 
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SUMMARY OF STUDIES EVALUATING COMPUTER-ASSISTED 
AND COMPUTER-MANAGED INSTRUCTION IN MILITARY TRAINING 
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Studies Of computer-assisted instruction (PLATO IV) at U S 
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Studies of computer-assisted instruction (PLATO IV) at 
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ST^S i5^°^^i^^^^ Operational Training Group, Pacific 
1975-1978. Management School, San Diego, California, 

Studies of computer-assisted instruction (PLATO IV) at 
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Studies of computer-managed instruction (Navy Computer 
Managed Instruction System) at Naval Air Technical Train- 
ing Center, Millington, Tennessee, 1975 . 
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studies of computer-managed instruction (Advanced Instruc- 
tional System) at Air Force Lowry Technical Training 
Center, Lowry APB, Colorado, 1978. 



TABLE D-1. STUDIES OF COMPUTER-ASSISTED INSTRUCTION (IBM 1500) AT U.S. 
ARMY SIGNAL CENTER AND SCHOOL, FT. MONMOUTH, NEW JERSEY, 1968-72 



Course 



Basic electronics 



Basic elect ron i cs 



Comparisons 



No. of 
Subjects 



Average 
Course 
Length, 
hrs. 



Conventional ^ 
CAI 



Conventional 
CAI 



278 
278 



11 
10 



11 
9 



Time Savings 
Compared to 
Conventional 
Instruction 
Hrs. % 



^Overall savings after 4 weeks CAI (line above) followed by 2 weeks 
conventional instruction for CAI and CI groups 



Performance 
Scores 



Attitudes 



Comnents 



References 



20 



Basic electronics 


Conventional 


155 


72 








CAI 


155 


50 


22 


30 


Basic electronics 


Conventional 


56 


102 








CAI 


56 


66 


36 


35 


Basic electronics 


Conventional 


142 


119 








CAI 


142 




43 


36 


Basic electronics 


Conventional 


142 


186 








CAI 


142 


139 


47 


25' 


*Two control groups; 


instructor, TV. 











Same 



Same on written; 
no diff. on writ- 
ten 4 perf. tests 
after additional 
week on CI 

CAI superior on 
performance test 

CAI superior on 
performance test 

Same on written i 
performance tests 

Same on written & 
performance tests 



Favorable to CAI 



Favorable to CAI 



Favorable to CAI 



Favorable to CAI 



Favorable to CAI 



Tutorial CAI. with IBM (1968) 
practical exercises 
on-line and off- 
line 



20 terminals 



22% less attri- 
tion in CAI group 

Attrition about 
the same' 

Attrition about 
the same' 



High-aptitude students on CAI save more time (14%) than 
low-aptitude students on CAI and also achieve better test 
scores (written and performance) 



Longo (1969) 



Giunti & 

Longo 

(1971a) 

Giunti & 

Longo 

(1971b) 

Longo 
(1972) 

Longo 
(1972) 
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TABLE D-2. STUDIES OF COMPUTER-ASSISTED INSTRUCTION (IBM 1500) AT NAVY RA<;Tr 
ELECTRICITY AND ELECTRONICS SCHOOL, SAN DIEGoI CALIFOrJiS: 1970-^^^^ 



Course 



Comparisons 



No. oi 



Average 

Course 

Length, 



Tinie Savings 
Compared to 
Conventional 
Instruction 
Hrs. % 



Performance 
Scores 



Attitudes 
Prefer CAI 



Alternating current 
theory 



Inductance 
Capacitance 
Direct current 

Series circuits 



Classroom 


200 


13.5-21. 


CAI 


51 • 


8.4-11, 


Classroom 


180 


17.0 


CAI 


50 


8.75 


Classroom 


64 


17.0 


CAI 


64 


7.6 


Classroom 


20 


10.3 


CAI 


50 


s.s 



Classroom 
CAI 



64 

50 



'Completion times for different ability groups. 
'Time savings for different ability groups. 



34.0 
17.7 



5-9^ 



8.3 



9.4 



4.8 



16.3 



33-44 



49 



55 



47 



48 



CAI superior 
(7S on test) 



CAI superior 
(lOS on test) 

CAI superior 
(lOS on test) 

CAI superior 
(3X on test) 



CAI superior 
(2X on test) 



CAI resulted in 
higher achievement 
at all abMity 
levels 



Prefer CAI 
Prefer CAI 
Prefer CAI 

Prefer CAI 



References 

Ford & 
Slough 
(1970) 



Huriock & 

Lahey 

(1971) 

Huriock 
(1972) 



Ford, 
Slough & 
Huriock 
(1972) 

ford, 
S1oug.h & 
Huriock 
(1972) 
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TABLE D-3. STl'OIES OF COMPUTER-ASSISTED INSTRUCTION (PLATO IV) AT U.S. ARMY 
ORDNANCE CENTER AND SCHOOL, ABERDEEN PROVING GROUND, MARYLAND, 1975 



Cowpdrlsons 



No. of 
Subjects 



Average 

Course 
length, 
hrs. 



NicMnlst course 
44E20 

; Nil Una (4 lessons ^ 



Uth< (7 lessons ) 



E v Tuition and 
Training materials 
developweflt~T7 

lessons ) ' 



Conventional 
CAl 

Programned 
text 

Conventional 
CAI 

Progranmed 
text 

Conventional 
CAI 

Progranned 
text 



354 

372 



115 
1B3 



66 

7o 



7.0 
1.3 
1.6 



B.O 
1.6 
2.8 



Time Savings 
Conpared to 

Co'..v-nt1ona1 
Instruction 
Hrs. : 



S.7» 
5.4 



9.8' 
9 2 



6.4' 
5.2 



Perfonnance 
Scores 



Attitudes 



_ Conwents 



References 



81 
76 



89 
84 



79 
65 



No significant 
deference 



No significant 
difference 



No significant 
difference 



Students and In- 
structors favor- 
able to CAI 



•Savings due to PLATO CAI vs. progranned instruction is 0.33/1.65 hrs (22t). 
^Savings due to PLATO CAI vs. progranned instruction is 0.60/1.77 hrs 
•Savings due to PLATO CA| vs. prograamed instruction is 1.12/2.77 hrs (40X). 



Basic conparison in 
study is bet)«een 
CAI and progranned 
instruction. See 
footnotes for those 
time savings. Im* 
provement of course 
material alone FY74- 
75 saved 0.49 hrs. 
(30X) for CAI and PI 
not included in data 
shown. 

PLATO not cost-effec- 
tive 

14 terminals 



U.S.Amty 
Ordnance Ctr 
& School 
(1975) 
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TABLE D-4. STUDIES OF COMPUTER-ASSISTED INSTRUCTION (PLATO IV) AT NAVY BASIC ELECTRICITY 
AND ELECTRONICS SCHOOL, FLEET AVIATION SPECIALIZED OPERATIONAL TRAINING GROUP 
PACIFIC FLEET AND MESS MANAGEMENT SCHOOL, SAN DIEGO, CALIFORNIA, 1975-1978 ' 



Course 



Comparison 



No. of 
Subjects 



Time Savinqs 
Average Compared to 
Course Conventional 
Length.* Instruction Perfonmance 
^^^s. Hrs^ Sep re s 



Attitudes 



Navy BE/E School* 
O scilloscope opera - 
tion (Navy Sonar A 
trainees) 



Multime ter (ohm 
i volt) 



Oscil losc ope 
operation ' 



North Island 
Naval Air 
Station' 

S-3 co«pilot INCOS ^ 
panel r-neration 



Actual oscillos. 29 
CAI simulation 20 



graphics 



Footnotes at end of table 



5.2 
5.2 



No difference No difference^ 



Actual gear 


42 


ID 








CAI /inter- 
active simu- 
lation 


42 


12 


-2 


-17 


No di fference 


Conventional 


18 


4.0 








CAI oractice 

CAI control 
functions 


12 
12 


4.1 

3.7 


-0.1 
0.3 


-3 

8 


CAI superior 
CAI superior 


CAI prompted 
practice 


12 


3.1 


0.9 


23 


CAI superior 


CAI measure- 
ment practice 


12 


3.3 


0.7 


18 


CAI superior 


Special ly 
developed 
work book 


22 


9.0^ 








CAI interactive 


22 


3.0 


6 


67 


CAI superior' 



Favor CAI 



Corrments 



References 



CAI simulation more 
effective than 
actual oscillo- 
scope for voltage 
& freq. msmt; less 
effective for man- 
ipulation of 
control s 

Addi tional time on 
CAI due to the 
novelty of the 
situation; students 
permitted to over- 
train themselves. 

Experimental study; 
only more effec- 
tive procedures 
would be recommended < 
all except CAI 
practice which does 
not save time 



Stern 
(1975) 



Lahey, 
Crawford & 
Hurlock 
(1976) 



Slough & 
Coady 
(unpubl , )" 



"CAI mere cost ef- Crawford. 

fective"(excluding Hurlock. 

cost of developing Padilla. & 

course materials) Sassano 
(1976) 

Study demonstrates Crawford & 

use of CAI graphics Crawford 

for simulation of (1978) 
operational equip- 
ment at low cost 

CAI group also com- 
pleted 2 tests in 
about 40v less time 
(each test less than 
5 min, ) 

( cont inued) 



ERIC 



TABLE D-4. (Continued) 



Course 



Comparisons 



Average 
Course 
No. of Length, 
Subjects hrs. 



Time Savings 
Compared to 
Conventional 
Instruction 
Mrs. X 



Perforroance 
Scores 



Attitudes 



Comments 



References 



Navy K«ss Hgmt. 
Specialist A School 

Recipe conversion 



Conventional 
CAI 

CAI/job aids 



20 
20 
20 



26.5 
4.3 
4.4 



22.2 
22.1 



84 

83 



No difference 
d 1 ( ference 



Favor CAI 



CAI effective for 
average & below 
average personnel 



Fredericks 
& Hoover- 
Rice 
(1977) 



'Student co'itact hours. 

'Basic electricity and electronics. 

'PLATO ^roup took 4 more minutes to complete test. 

*S€e Wurlock and Slough (1976), p. 21-22. 

*Ircegrated Control System. 



*Mean number of problems completed in high fidelity position trainer. 
'Eight hours workbook plus 1 hour on position trainer. 
•Navy Sonar A School (controls). Navy BE/E School (experimental). 
'Fleet Aviation Specialized Operational Training Group, Pacific Fleet. 
Compared to work book. 
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TABLE D-5. STUDIES OF COMPUTER-ASSISTED INSTRUCTION (PLATO IV) AT AIR FORCE 
SCHOOL OF HEALTH CARE SERVICES. SHEPPARD AFB. TEXAS. 1977 



Course 



Physician Assist- 
dnt Course 



Cofflpdri son 



Average 
Course 

NO. of Length, 
Subjects hrs. 



Time Savings 
Compared to 
Conventional 
Instruction 
Hrs. 



Performance 
. Scores 



Attitudes 



Comments 



References 



Respiratory sys- 
terns and illness ; 
cMnical problem -. 
soivinq skills 



Radiology 
Dental assistant 



Lecture 
CAI 



Medical laboratory Lecture 
CAI 



Prog, text 
CAI 

Lecture 
CAI 



154 

93 



S9 
97 

28 



18. 5» 
16. 0> 



9.0' 
7.S> 



4.5 
4.0 

8.0' 
7.0 



'Includes 4.5 hrs. review for CAI and lecture groups. 
'Experimental POMC (problem-oriented medical curriculum). 
'Calculated from data in report. 
* Compared to Proqranwed text. 



2.5 



14 



1.2 



14 



0.5 ir 



1 13 



No significant 
di f ference 



CAI 201 superior 



CAI 31 superior 



No difference 



Students negative, 
worried about 
falling behind 
those in tradi- 
tional lecture 
(this judgment 
was incorrect) 

Prefer CAI 



Prefer CAI 



Prefer CAI 



Preliminary findings Steirikerch- 
only; most results ner. 
inconclusive. Deignan. 

Limited CAI material; [{^^^''^* * 

16 hrs spread over 

13 weeks. ^ ^1977) 



16 terminals. 

CAI saves more time 
and yields greater 
achievement at high 
and low aptitudes; 
progranmed instruc- 
tion favors high S 
medium aptitudes. 



Deignan 

Duncan 

(1977) 



Deignan & 
Duncan (1977) 



Preliminary findings Deignan 
(no date) 
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TABLE D-6. STUDIES OF COMPUTER-ASSISTED INSTRUCTION (PLATO IV) AT AIR FORCE 
CHANUTE TECHNICAL TRAINING CENTER. CHANUTE AFB. ILLINOIS, 1977 



Course 



Comparisons 



Averaoe 
Course 

No. of Length, 
Subjects hrs. 



Tine Savinas 
Compared to 
Conventional 
Instruction 
Hrs. 



Special purpose 
vehicle repairman 



Crash/ fire vehicles Conventional* 



PLATO* 



Refuelling vehicles Conventional* 
PLATO* 



38 



13 



35 
37 



236 



145 



233 
166 



91 



67 



39 



29 



H>terlal handling 
vehicles 



Conventional* 

Unvali dated 
TLATO 

PLATO* 



57 
30 

21 



236 
233 

173 



63 



27 



f'ootnotes at end of table. 



Performance 
Scores 



Attitudes 



Comments 



No significant dif- Students prefer 
ference between PLATO PLATO to pro 



and conventional 
groups in field per- 
formance ratings 6-12 
mos. after gradua- 
tion, (PLATO N«119; 
conventional N-lOl.) 
All other data based 
on performance scores 
at school. 

No significant 
difference 



grammed instruc- 
tion 



PLATO not cost 
effective. 



No signi ficant 
di f ference 



Instructors favor- y No advantage to 
able to PLATO at / unvalidated CAI 
first but became I materials, 
dissatisfied later. p^„g^^^, 

istnjction saves 
time;(amount not 
stated) PLATO 
saved lOt- more. 
3D terminals 

' No advantage to 
unvalidated CAI 
materials. 

I Progranned ins.*«nic- 

tion sav*? ■ ... 
j (amount n 

' PLATO sa^ ' -C? r^r ■ 
30 termina?; 



No significant 
difference 



Instructors favor- 
able to PLATO at 
first but became 
dissatisfied later. 



No ad vi.*; io 
unval1d;;t:^*d CAI 
.materia «;v 

'Progrt red 5rttruc- 
itlon St 7c^. t (•;><«; 
'(amount, r:*. -i^au^l 
PLATO s.».>'^ iCi r i 

l30 term. m> 



References 



Dallt'dn, 
OeLeOiMaIn 
& Ginmen 
(1977)' 



Dal 1 man, 
DeLeo, Hain 
A Glllman 
(1977) 

D&llman 
DeLeo, Main 
& Glllman 
(1977) 

OalUian, 
DeLeo, Main 
& Gillinan 
(1977) 

w. Hair 

& v.'llman 
(1977) 



Oallman, 
DeLeo, Main 
' Glllman 
1977) 



(continued) 
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TABLE D-6, 



(Continued) 



Course 


Comparisons 


No. of 

Subjects 


Average 
Course 
Length, 
hrs. 


Time Savings 
Compared to 
Conventional 
Instruction 
Hrs. % 


Performance 
Scores 


Attitudes 


Comperyi:- 


References 


Towinq and Service- 
\ng vehicles 


Conventional^ 

Unval1ddte<l 
PLATO 

PLATO- 


50 
14 

41 


227 
234 

174 


53 23 


No significant 
d1 ff erence 




Abjye coiMcnts 
app'y. 

So Si 9..'. Meant zf- 


Dallmani 
DeLeo.Kaln. 

& G1i;:ti^)^ 
(1977) 



,oct attrition 
r^vei <• ? -month 



Ooly Blocks 1 - 4 (cowwon course segment shared by all four courses) were Instructed 
by PLATO IV. 

*Data collected prior to PLATO study. Apr. 1974 - 15 Jan. 1975; Includes prograitred text. 
*0«ta collected during PLATO study, group paced. 15 Jan. 1975 - 30 Sept. 1975. 
'See Hlffiwich (1977) for a critique of the Chanute study. 

"Validated so that failure rate not larger than "^O percent on Master Validation 
Exam with sample of 20-30 students 
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TABLE D-7. STUDIES OF COMPUTER-ASSISTED INSTRUCTION (LST-3M AT AIR FORCE KEESLER 
TECHNICAL TRAINING CENTER, KEESLER AFB, MISSISSIPPI , 1972-73 ■ 



Course 



Comparisons 



Ko. of 
Subjects 



Rgsistive. capacatlve 
and inductive c<r- 
cults (week 5 in air 
traffic control stand- 
ardized electronics 
principles course) 

Weather (part of air 
traffic control 
operator course) 



Resi stive, capa- 
cltlveandinduc - 
tiye circuits (week 
5 In air traffic 
control standard- 
ized electronics 
principles course) 



Resistive, capa- 
citive and induc- 
tive circuits (week 
5 In air traffic 
control standard- 
ized electronics 
principles course) 



Conventional ^ 
Conventional ' 
CAI 

Conventional ^ 
Conventional ' 
CAI 



Conventional ' 
Sel f-paced 
CAI 



Conventional ; 

Low ability 
^ Hedium " 
Higher " 
Highest " 

CAI-1n<!1vidua1 : 



55 
55 
55 

22 
22 
20 



55 
30 
30 



12 
12 
14 
17 



Average 
Course 
Length, 
hrs. 



30 
30 
19 



16.0 
16.0 
13.4 



30.0 
16.4^ 
12.3- 



30.0 



Time Savings 
Compared to 
Conventional 
Instruction 
Hrs. ^ I 



11 



2.6 



13.6 
17.7 



Perfortnance 
Scores 



Attitudes 



Corments 



References 



37 



16 



45 
59 



Low ability 


10 


22.4 


7.6 


25 


Medium " 


17 


18.8 


11.2 


37 


Higher " 


19 


18.0 


12.0 


40 


Highes. 




17.6 


12.4 


41 


CAI-paired*; 










LoM ability 


12 


25.9 


4.1 


14 


Medium " 


12 


18.2 


11.8 


39 


Higher " 


12 


13.4 


16.6 


55 


Highest " 


12 


12.7 


17.3 


58 



No significant 
differences at 
end of course or 
6 weeks later. 



CAI lower than con- 
ventional but not 
statistically 
significant 



Favorable to CAI 



Favorable to CAI 



)CAl superior 



Favorable to CAI 



Higher ability 
groups achieved 
higher scorer ';ut no 
significant differ- 
ences between method 
of training; ability 
level of student in 
a pair does not ef- 
fect other's score. 



Favorable to CAI, 
especially when 
paired. 



Students in CAI re- 
lied on Instructor 
only 2? of time; 
lesson material 
tested but not vail 
dated prior to test.* 



Harris, 
Grossberg, 
Downs et al. 
(1972). 



Students in CAI re- Downs, 
Tied on Instructor Johnson, 
only 2% of time; Barnes et aU 
lesson material (1972) 
tested but not vali- 
dated prior to test. ' 

lesson materials 
judged not entirely 
adequate. 

Students in CAI re- Keesler AFB 
lied on Instructor (1972) 
only 2% of time; les- 
son material tested 
but not validated 
prior to test.' 

High aptitude subjects 
(upper 20X)on1y in both 
groups (excluded in pre- 
vious tests) 



Paired students on 
CAI save more time 
than those working 
alone, except for 
the low-ability 
group. 



Keesler AFB 
(1973) 



Footnotes at end of table 



(continued) 
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TABLE D-7. 



(Continued) 



Course 



Comparisons 


No. of 
Subjects 


Average 
Course 
Length , 
hrs. 


Time Savings 
Compared to 
Conventional 
Instruction 
Hr5. I 


Conventional ^ 


55 


30.0 




Progranned text 


66 


12.8 


17.2 57 


progranned text 


106 


16.5 


13.5 45 


CAI' 


55 


19.0 


11 37 



Performance 
Scores 



Attitudes 



Cotments 



References 



Reslstlve.capa- 
citlve and induc- 
tive circuits (week 
5 In air traffic 
control standard- 
ized electronics 
principles course) 



programmed text and 
CAI better than con- 
ventional but no 
practical signifi- 
cance 



Favorable to 
progranned text 
and CAI 



Quality of train- 
ing material (pro- 
granned text) more 
important for 
achievement and time 
savings than presen- 
tation mode (CAI) 

Differences in time 
savings between pro- 
granned text and LTS 
not significant 



Keesler 

AFB 

(1974) 



^Students aware of role as control group. 
^Students not aware of role as control group. 

'Lower aptitude students spend more time in training; correlation general AQE 
score and time spent in training is -.30 (p < .05) Butman and Frick (1972). 

^Computed from statement in paper that self-paced group was 45.4 percent faster 
th*n conventionally taught students and LTS saved 25.3 percent over self-paced 
*Two students at one terminal working jointly. 
'Lincoln Terminal System-3. 

'Data from Harris, Grosberg, oowns <c al, (1972) above. 



group. 
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TABLE D-8. 



^Ku^^f.^^"''"''"^'^-'^^^^^!"^'' INSTRUCTION (TICCIT) AT NAVY SQUADRON VS-41 
NORTH ISLAND NAVAL AIR STATION, SAN DIEGO, CALIFORNIA. 1978 



Course 



. CoiwpTisons 



Average 
Course 

No. Of Length, 
Subjects hrs. 



Tine Savings 
Conpared to 

Conventional 
Instruction 
Hrs. % 



Tactical coo rdinator 
S-3A ASU aTtcTiyt — 



Workbooks 
CAI 



0.67 
0.38 



Perfortttnce 
Scores 



Atti tudes 



Cownents 



References 



No difference 



0.29 



43» 



No difference In 
attitude and con- 
fidence ratings. 



'Compared to work book. 



Experiment limited 
to 1 lesson (under 
1 hr. ); 5 months 
needed to collect 
data; In addition 
to training timet 
CAI group also saved 
42t time In taking 
test. 



UfiUer 
(1978) 
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TABLE D-9. STUDIES OF COMPUTER-ASSISTED INSTRUCTION (IDIIOM AND PLATO IV) AT 
NAVY GUIDED MISSILE SCHOOL, DAM NECK, VIRGINIA, 1975 



Course 



Comparisons 



No. of 
Subjects 



Average 
Course 
Length, 
hrs. 



Time Savings 
Compared to 
Conventional 
Instruction 
Hrs. t 



Perfonnance 
Scores 



Attitudes 



Cownents 



References 



Fire Control Tech- 
nician Replacement 
Training 

Integrated Test 
Operatinii Panel 



KeYboard Subsystcfn 



Conventional 
IDIIOM^ 
PLATO IV 

Conventional 
IDIIOM^ 
PLATO IV 



35 
13 
18 

38 
19 
17 



2.2 
1.9 
2.9 

3.0 
1.9 
2.6 



0.3 
-0.7 



1.1 
0.4 



14 

-32 



37 
13 



Written: no diff. 
perfonnance: 

CI.PLATO:no diff. 

CI superior to 
IDIIOM 

No difference 



Favorable to CAI 
except S% 
unfavorable. 



TICCIT included in 
study plan but in- 
sufficient data 
precludes evalua- 
tion. 



General El- 
ectric Ord- 
nance Sys- 
tems, 
(1975) 

Radsken and 

GrossoR 

(1975) 



Mnformation Displays, Inc. Input-Output Machine, a stand-alone CAI system. 
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^'^^'■^ O*" COMPUTER-MANAGED INSTRUCTION (NAVY COMPUTER MANAGED INSTRUCTION 

SYSTEM) AT NAVAL AIR TECHNICAL TRAINING CENTER, MILLINGTON, T'ZNNESSEE, 1975 



Course 



Aviation 
Fianiarlzation 



Aviation 
^aalllarliatlon 



Aviatl^on Mechan» 
leal Fundajnentals 



Aviation Mechan- 
ical Fundamentals 



Comparisons 


No. of 
Subjects 


Average 
Course 
Length, 
hrs. 


Conventional 


142» 


33.6 


Instr. managed' 


142 


10.3 


Comp. managed 


142 


10.5 


Conventional 


99* 


34.4 


Ins tr. managed' 


99 


10.3 


Comp. managed 


99 


11.7 


Conventional 


142» 


73.2 


Instr. managed' 


142 


38.3 


Comp. managed 


142 


39.7 


Conventional 


99* 


70.3 


Instr. managed' 


99 


41.3 


Comp. managed 


99 


41.0 



Tine Savings 
Compared to 
Conventional 
Instruction 
Hrs. X 



Performance 
Scores 



Attitudes 



CoBinents 



References 



23.3 
23.1 

24.1 
22.7 

34.9 
33.5 

29.0 
29.3 



69 
69 

70 
66 

48 
46 

41 
42 



Slightly better 
than conventional^ 



t Better than con- 
jventional^ 



\Better than con- 
jventional^ 



j Better than con- 
jventional* 



No difference 
betvreen methods 



No difference 
between methods 

No difference 
between methods 



No difference 
between methods 



Conv<rntional course 
revised for this 



Carson, 
Graham, 



study saved 50% time Kardinq, 
of previous course, et al.(1975) 

Carson, 
Graham, 
Hardina, 
et a:. (1975) 

Carson, 
Graham, 
Harding 
et al.(1975) 

Carson, 
Graham, 
Hardinq 
et aZ.(1975) 



*Navy students ( ''minimum number"). 
^Marine students ("minimum number"). 

'Same course as computer-managed, except that instructors 
graded exam'nations and assigned lessons. '"structors 



"•Statistically significant on end of course test and shop grade; no 
difference at end of school or 6 weeks later; no difference in 
attrition. 
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TABLE D-11. STUDIES OF COMPUTER-MANAGED INSTRUCTION (ADVANCED INSTRUCTIONAL SYSTEM) 
AT AIR FORCE LOWRY TECHNICAL TRAINING CENTER, LOWRY AFB, COLORADO, 1978 



Course 



No. of 

_Coiiipar1sons Subjects 



Average 
Course 
Length » 
Days 



Time Savings 
Compared to 
Conventional 
Instruction' 
Days X_ 



Performance 
Scores 



Inventory Manage- 



Conventional ^ 

AIS* 2565 



Material Fac^ities Conventional* 

AIS 634 



Precision 
Neasurinq 
tquipwent ' 



Conventional* 
AIS 



602 



Weapons Mechanic Conventional* 

AIS 2745 



Inventory 
ManaqetnelTt 



Materiel 
FaciUHes 



Precision 
Measuring 
i^uipwent^ 

Weapons 



pon 
Mecnanic 



MSP* 
AIS 



MSP 
AIS 



MSP 
AIS 



MSP 
AIS 



49.5 
39.0 



43.5 
33.1 



229.0 
204.0 



94.0 
49.8 



34. d 
30.4 

24.7 
24.7 



89.9 
92.8 

40.6 
42.1 



10.5 



10.4 



25 



44.2 



24 



47 



4.4 13* 



-2.9 



-1.5 -4* 



No difference 



No difference 



No difference 



No difference 



No difference 



No difference 



No difference 



No difference 



Attitudes 



Comments 



Students favor- 
able, instruc- 
tors neutral or 
unfavorable 



Students favor- 
able, instruc- 
tors neutral or 
unfdvarable 



Students favor- 
able, instruc- 
tors neutral or 
unfavorable 



Students favor- 
able, unstruc- 
tors neutral or 
un/avorable 



Data for 12 mos. 
ending 28 Feb 
1978 



Data for 12 mos, 
ending 28 Feb 
1978 



Dfita for 12 mos. 
ending 28 Feb 
1978 



Oa.a for 12 mos, 
endinq 28 Feb 
1978 * 



Data for AIS Ser- 
vice Test, Feb- 
Ouly 1978 

Average course 
lenqths not di- 
rectly comparable 
to data above be- 
cause reported 
differently. 

Only limited seg- 
ments of PHE 
course used in 
test. 



M974, prior to AIS. 

'Advanced Instructional System. 

'li?* "^^"9^ corrected for changes in plan of 
PJ€ course was a vacuum-tube oriented, AIS course 
electronfcs oriented. 



_References 



Briefing 
material 
AFHRL-n, 
Lowrv AFB 
April 1978 

Briefing 
material 
AFHRL-TT, 
Lowry AFB 
April 1978 

Briefing 
material ; 
AFHRL-TT, ' 

Lowry AFB 
April 1978 

Briefing 
material 
AFHRL-n, 
Lowry AFB 
April 1978 

Briefino ma- 
terial AFHRL- 
TT, Lowry AFB 
Sept 1978 

Briefing Ma- 
terial AFHRL- 
TT, Lowry AFB 
Sept 1978 



Briefinq ma- 
terial AFHRL- 
TT, Lowry AFB 
Sept 1978 

Briefing ma- 
terial AFHRL- 
TT, Lowry AFB 
Sept 1978 



instruction, 

is transistor/digital 



^Manually Self.paced (MSP). Students in both groups used some AIS course 
^ r ! , J" ^ f'^"^* instructors scored tests manually and guided 
students without b;,ing AIS terminals. yuiueu 

^Compared to Manually Self-paced, not conventional instruction. 
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APPENDIX E 



SUMMARY OF DATA ON COSTS OF INSTRUCTION 



This appendix presents the data on costs of instruction 
that we were able to find and which are summarized in Chapter 
IV. The presentation is organized according to type or func- 
tion of resources required to conduct military training. These 
categories are identified in Table E-1. 

The data are shown as reported in the literature. No ad- 
justments have been made to bring cost levels to a common base 
period. The time periods in which costs were incurred are 
generally not shown in the source documents and may differ, by 
varying periods, from dates of publication. The use of standard 
indices, such as wholesale prices, does not appear appropriate 
to adjust all costs to a common base; specialized indices, that 
are not available, would be required for some types of resources, 
such as various components of computer systems. 

The validity of individual data has not been evaluated. 
Some data values were extracted from secondary sources that did 
not reference original sources. Data in some secondary sources 
duplicated information already available in primary sources and 
were not used; however, undetected duplications may remain. Data 
that were not well enough described to be interpreted with confi- 
dence have been excluded. Wherever a value was shown, it was 
assumed to be based on historical experience unless it was 
specifically described as a programmed or planned value. 

No references to costs of conventional instruction were 
found. This may be due to our approach to the literature. 
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TABLE E-1 . RESOURCES REQUIRED FOR VARIOUS METHODS 
OF INSTRUCTION IN MILITARY TRAINING 



Resource (Type or Function) 


See 
Sec ti on , 

This 
Append i x 


Program Development 
Program Design 

Instructional Materials:^ Conventional Instruction 

Individualized Instruction 
P rog rammi ng 

Fi rst Uni t Production^ 
Computer-Based Instruction 
Programmi ng 
Coding 


1 ^ 


2* 


3 


4 


Program Del i very 

Instructional Support Personnel^ 
Equipment and Services:^^ Laboratory (including simulators) 

Media Devices 

Computer Systems 

Communi cations 
Materials (including consumables)^ 
Faci 1 i ti es^ 


5 


6* 


7 


8 


9 


10* 


1 1 


'^rogram Management and Administration 


12* 


Student Personnel: Pay Allowances 

Other (PCS. TDY . etc. ) 


1 3** 



NOTE: * No data available. 

** Included in discussion of Instruction; Instructors and Instructional 
Support Personnel (Section 5). 

^Includes revision/ 
^Master copy. 

^All direct personnel not included in other categories. 

^Includes all hardware-related costs: initial (including installation and checkout), 
modification, and replacement; operation and maintenance, lease and user fees; computer 
system software; etc. 

^Incluoes copies of instructional materials (books, courseware copies, etc.). 

^Structures, fixtures, and furnishings. 
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Emphasis was placed on computer-based instruction, and we ex- 
pected that it would be evaluated in terms of its alternatives 
(conventional or individualized instruction). This was not the 
case. Student hours required by computer-based and conventional 
instruction were frequently compared; however, these data were 
not converted to equivalent dollar costs. Other resources needed 
in computer-based instruction were simply presented in dollar or 
real terms, but no other data wri e reported on the costs of con- 
ventional instruction. This raises an obvious question. V/hat 
good does it do to know the cost of some particular version of 
computer-based instruction if little is known about the cost of 
conventional instruction or of any other method of instruction 
to which it might be compared? 

Several studies noted that significant man-hours are asso- 
ciated with Program Design, but provided no further information. 
Several studies noted that savings, due to decreases in student 
hours, represented the combined impact of course revision and a 
change in the method of instruction. One study noted an expen- 
diture of in man-years for a course revision that decreased the 
length by 50 percent but provided no information regarding what 
was involved in the revision. The magnitudes of these values 
amount to a strong argument for considering the benefits of 
course revision alone, without changes in instructional method. 
The cost-effectiveness of course revisions alone should be eval- 
uated as a competitor of CAI , CMI , and individualized instruction. 

No data nor any discussion was found regarding Instruc- 
tional Program Management requirements, and only one was cited 
that discussed o uer Instructional Support Personnel resources. 
This can be unde. stood with regard to CAI where all applications, 
save one, have t . >n experimental programs of limited duration. 
However, with respect to other instructional methods, including 
CMI, it shoulc have been possible to develop such information. 
With the highly or-anized structure of military training, one 
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must allow that both Program Management and Other Instructional 
Cupport may account for a significant share of total program 
cost, that they are subject to analysis, and that they may vary 
sufficiently between different methods to have a noticeable im- 
pact on cost-effectiveness. 

E.l PROGRAM DEVELOPMENT: PROGRAM DESIGN 

No data were found on these costs of instruction. 

E.2 PROGRAM DEVELOPMENT: INSTRUCTIONAL MATERIALS FOR 
COVENTIONAL INSTRUCTION 

No data were found on these costs of instruction. 

E . 3 PROGRAM DEVELOPMENT; INSTRUCTIONAL MATERIALS FOR 
INDIVIDUALIZED INSTRUCTION 

Development of instructional materials for individualized 
instruction involves two distinct func'*:ions that should be kept 
separate. The first is the Instructional Programming (ur author- 
ship). The second is the First Unit Production 'or master copies) 
of the courseware material. Only two of six studies reporting 
costs maintained this distinction. The most notable feature of 
these data is their wide range. Table E-2. For naster copy pro- 
duction the variability holdr both betv;een different types of 
media material and within one type. Depending upon the number 
of silent slides cr printed pages that might comprise an hour of 
instruction, the data indicate a possible range between a few 
hundred dollars to over $10,000 per hour of instruction. The 
U.S. Army Ordnance Center and School (1975), the single source 
of information on authoring requirements, noted values of ^0 
man-hours per instructional hour for sound-slides and 280 man- 
hours per instructional hour for sound motion picture or TV. 
These limited data and the limited discussions presented in the 
citations make it impossible to understand the reasons for the 
difference. Table E-3 displays the only information found on 
courseware material reliability, and no information was provided 
as to the repairability or repair costs of failed courseware. 
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TABLE E-2. 




Notes: 



2) 
3) 

4) 



5) 
6) 



instructional support A« i.I.miJ^"*": l««»on should bt dts1an«d t« h- < ^ 



Place $25 per sude 
if tnrtc and twelve nan* 
percent of courseMare 



Th. ^ -rr.*/Kriai« COKpat aPfllS SChOOl . --^-eiui/we 

tH.t tH. tot., .„o.t ... ;;voj;r?s'j„n.:jt?s„*.r ^os;?::?:;' "Sur:?'?:? -f.c:;:r„'r:.2i.'n'';u;? 
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TABLE E-3. MEDIA COURSEWARE RELIABILITY 



Materi al 



Uses 



Failures 



Super 8 Film 



1 ,548 



7 



Video Tape 



982 



1 8 



F i 1 m s t r i p 



24,445 



221 



Audio Tape 



25,1 54 



36 



Source: AIS Integrated System lest, 
October 1977. 



A variety of media are available for presentation of indi- 
vidualized ins '.ruction, and different media are substitutes for 
each other in presenting the sub.lect matter of small units of 
instruction, e.g., the individual lesson. As a result, a course 
(or segment of a course) may utilize a mixture of media. With 
the variations in courseware costs noted, different mixes of 
meaia can imply sizeable differences in course costs. However, 
determination of the most effective media i-nix requires a ra+-.her 
ext.--nsive course design effort, and systematic investigation of 
: L^.ernative course designs to determine cost differences asso- 
ciated with these mixes is an expensive process. 

E.4 PROGRAM DEVELOPMENT: INSTRUCTIONAL MATERIALS FOR 
C'HPUTER-BASED INSTRUCTION 

Table E--I4 displays information on development of computer- 
based instructional materials. Close to 1,000 hours of materiel 
are represented, but in only three cases were more than ^0 hours 
produceu by one authoring group. The striking feature of the 
table is the variability of the data—approaching an order of 
magnitude . 

Several studies provided data on man-hours needed sepa- 
rately for authoring and coding. For CMI, authorship was the 



dominant requirement and accounted for 90 percent of the man- 
hours. For CAI, weighted averages indicate that the require- 
ments for each function are roughly equal, with 52 percent of 
the total attributed to authoring. However, wide differences 
may be noted with authoring accounting for as little as 11 per- 
cent and as much as 75 percent. 

Instructional materials may be developed in-house by 
military or civilian personnel or by contract. Military per- . 
connel would coi-e from the more senior pay grades, e.g., E-5 or 
E-6) with pay and allowance rates near $6 per hour. Costs of 
contract personnel should be roughly $30 per hour (Middleton, 
Papetci, and Micheli, 197^ adjusted to 1978 wage levels). On 
the basis of these hourly labor rates and the man-hours require- 
ments shown in Table E-H, the costs of courseware development 
might currently be estimated as high as $21,000 or as low as 
$500 1 or instructional hour for CAI and as high as $3,300 or as 
low as $?00 for CMI. 

Grimes (1975) presented the only analysis of programming 
requirements. He cites 80 man-hours per instructional hour as 
the weighted average of 16 programmers, all of whom were either 
students or project personnel at the University of Illinois. On 
an individual programmer basis, man-hour expenditures per in- 
structional hour averaged 182. The difference in averages in- 
dicates there are great differences in individual productivity. 
The eight most "productive" programmers developed 239 hours of 
instruction, expending a weighted average of 56 hours; the eight 
least productive programmers developed 76 hours of instruction 
with a weighted average expenditure of 157 man-hours; at the 
extremes, one programmer spent 1,389 man-hours to produce one 
hour of instruction while another programmer produced 3k in- 
structional hours with an average expenditure of 31 man-hours 
per instructional hour. 
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TABLE E-4. 



TIME REQUIRED TO DEVELOP INSTRUCTIONAL 
MATERIALS FOR CAI AND CMI 



Method of 
Instniction 




Instnictional 

Hours 
Developed 


Man-hours Per Instructional Hour 




See 
Note 


Hardware System 


Programming 


Coding 


Total 


Reference 






39.0 


27 to 248 


50 to 467 


77 to 714 


Hiirlock Er Slough. 1976 


1 






6.0 
2.0 






156 
400 


Kribs. 1976 
Kribs. 1976 


2.8 




PLATO IV 


30.0 
20.0 






284 
100 Er 200 


U.S. Army Ordnance Center Er School, 1975 
Oallman. Oeleo, et aL, 1977 


3 






32.0 


141 


81 


222 


Himwich, 1977 




CAI 




315.0 






80 


Grimes, 1975 


4 




TICCIT 


100 
3.0 






200 
400 


Kribs, 1976 
Kribs. 1976 


2,8 






32.0 


150 


96 


246 


Himwich, 1977 






LTS3 


30.0 






175 


Keesler. 1973 






IBM 1500 


3.5 


356 


119 


475 


Rogers Er Weinstein, 1974 


5 




Unspecified CAI 


Unknown 






150 Er 200 


Middieton, Papetti, Er Micheli. 1974 


6,8 






50.0 


100 


10 


110 


Carson, Graham, et aL. 1975 




CMI 


Navy CMI 


300.0 






30 Er 60 


Hansen. Ross, et aL. 1975 








Unknown 


12931 


(25) 


(318) 


Polcyn, Baudhuin, et aL,. 1977 


i 



3-26-79-32A 
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Notes: 

1. The ranges for Programming (27 to 298). Coding (50 to 467) and Total (77 to 714) man-hours aie the extreme values experienced in eight experime^ita^ 
applications. 



liBtructionil Hours 
Oeveioped 


NUn-hours Per Instnictionil Hour 


Authoring (S) 


Coding (S) 


Total 


88 


27i35) 


50 (65> 


77 


52 


115(441 


146 (56) 


262 


)3 


173 I&41 


97 (36) 


270 


120 


158 1511 


150 (49» 


308 


30 


167 143^ 


217 (57) 


383 


30 


W (111 


427 (89) 


480 


4.0 


107 (19) 


453 (81) 


560 


20 


248 i35> 


467 (65) 


714 


Total 39 2 
Average 


131 I39» 


251 (66) 


382 



2. The man-hour rales of 400 reported for both PLATO IV and TICCIT appear to be the initial segments of courseware also reported in Himwich. 1977. 

3. The difference is attributed to difficulty of the material programmed. In the case of the 100 man hours per hour value, the courseware is described only 
as "simple". 

4. The value of 80 man hours per hour is a weighted average. Sixteen programmers were involved to differing extents in developing 315 instructional hours 
with a total expenditure of 25.000 man hours. On an individual programmer basis their average man hour expenditures ranged from 31 to 1.380 man hours per 
instructional hour and averaged 182. In the case of two authors working in the experiment over a four year period the time required to program an hour of 
instruction in the ihrid and fourth years averaged 45 percent and 15 percent less than that required in the initial two years. 

5. This report concerns a civilian higher education application and Is written in a very laconic fashion. The values for both instructional hours and man-hours 
expended were derived from other information contained in the report and may be subject to wide error of interpretation. The value of 475 man hours per 
hour includes only the hours attributed to "Authors b Instructional Oesigners" and "Computer Programmers". However, other functions (such as curriculum 
analysis) appear to be included. 

6. The difference in values is attributed to the level of experience of authors. The study asserts that annual revision requirements stand at 10 percent to 12 
percent of initial requirements, but no source or back up material is presented. In addition, an hourly rate of $21 is given for commerical courseware develop 
menl based on a direct labor rale of $8.25 with overhead of 100 percent, general and administrative expenses of 15 percent, and profit of 10 percent. Again, 
no source or back up is presented. 

7. Requirements were given in terms of dollars per hour (authoring at $2,390 and coding at $250) and have been converted on the basis of $10 per man hour 
for military labor. 

8. Secondary source. 
3 6-79 2 
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In these data, there Is no evidence of differences between 
students and project staff, and there Is nothing In the Infor- 
mation presented to tie man-«hours expended with difficulty of 
either the material programmed or the Instructional approach. 
In one case, two student programmers were employed for about 
^ years and produced a total of 97 hours of course materials. 
Comparison of the productivities for the first and second 2-year 
periods presents evidence of appreciable learning. 

The values of man-hours per Instructional hour given by 
Hansen, Ross, Bowman, and Thurmond (1975) are so much lower than 
other reports that they are Immediately suspect. They are based 
on the three courses converted to CMI at the Memphis Naval Air 
Training Station and represent roughly 300 hours of CMI mate- 
}Mals. The value of 60 man-hours per Instructional hour was 
derived from a survey of personnel participating In the programs. 
However, the sample was extremely limited; of 13 Individuals 
polled, only five provided quantitative answers, and these ranged 
from 10 to 150 man-hours per Instructional hour. The value of 
30 man-hours per Instructional hour Is presented as a "currently 
estimated" requirement Including "textual media conversion as 
well as computer activity" but does not reference the author 
survey or another source. 

E.5 PROGRAM DELIVERY: INSTRUCTION, INCLUDING INSTRUCTORS AND 
INSTRUCTIONAL SUPPORT PERSONNEL AND STUDENT PERSONNEL 

Studies of computer-based training typically define de- 
creases In course durations as an element of training effective- 
ness. HoweVer, time spent by military personnel In any function 
Is a cost, since personnel received resource support In the form 
of salaries, housing, eta. Of the several studies that addressed 
both training time and training cost, only four attributed costs 
to student and Instructor time. 

Many of the studies reported savings due to reduced times 
required by students to complete courses with computer-based 
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Instruction. Since computer-based Instruction also requires 
some additional resources, such as for the computer. It Is 
obvious that It Is not clear that there are overall savings un- 
less both values are expressed In comparable terms, i.e., dollars. 

The omission In translating student and Instructor time to 
cost Is Inconsistent with the treatment generally afforded to the 
cost of developing Instructional materials. In most cases, the 
courseware was developed by military personnel, and their costs 
were attributed to the cost of the courseware. There are no 
grounds for distinguishing these personnel costs from other per- 
sonnel costs thit were not Included, e.g.^ Instructors and stu- 
dents . 

The cost Information developed In three of the four studies 
Is shown In Table E-5. The $61,000 pay and allowance rate shown 
for Crawford, Hurlock, et at. ^ (1976) Is described as the "billet 
cost for the lowest ranked student or Instructor" (an aviation 
lieutenant). It Is over two times the standard pay and allow- 
ance factor associated with junior flying officers and Includes 
a variety of personnel support Items over and above those, In- 
cluded In pay and allowances, e.g',, command and administration, 
medical costs, dependents' schools, travel, and retirement. 

The full effect of computer-based Instruction on personnel 
costs Includes Its Impact on requirements for Instructors and 
other types of direct support personnel, and computer-based 
Instruction Is generally attributed with allowing Increases In 
student: Instructor ratios. From the Information shown In Table 
E-5, this Increase does not appear to have a relatively signifi- 
cant effect on cost. Development of Instructional materials Is 
a significant cost associated with the Introduction of computer- 
based Instruction, and the extent to which changes In student: 
Instructor ratios offset Its cost depends upon the number of 
students receiving a given hour of Instruction* Based on the 
sketchy Information provided by Table E-5 (assuming an expenditure 
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TABLE E-5. INSTRUCTOR AND SUPPORT COSTS FOR PROGRAM 
DELIVERY, COMPUTER-BASED INSTRUCTION 





SOURCE 


DATA ITEM 


Carson, 

Uiallalll, 

Harding 

etaL 

(1975) 


Hansen, 
Ross, 

Bowman, & 

Thurmond 

(1975) 


Crawford, 
nunocK, 
Padillo, & 
Sassano 
(1976) 


System 


Navy CMI 


Navy CMI 


PLATO IV 


Change in Ratios 
Students : Instructors 

Students to Instructional 
(Direct) Support 

Students to Indirect 
(Base) Support*) 


10:1 to 
16:1 


7.5:1 to 
9.0:1 

Unchanged 
at 24:1 

Unchanged 
at 12.5:1 


a 


Pay and Allowance Rates 
Students 
Instructors 
Instructional Support 
Indirect Support 


$5,899 
9,697 


$ 5,300 
10,800 

72,400 


$61,000^ 
61,000C 



'Cannot be expressed in these terms. The net result was to eliminate the single instructor-hour .ontained in a 9-huur 
training segment. 

^Applies to students, instructors, and instructional support perionnel. 

i^The $61,000 figure is described as billet cost and includes a variety of personnel support ilii.iS over and above 
pay and allowances, a^.. command and administration, dependent school costs, recruitino costs, reenlistment 
bonuses, and retirement; students and instructors were pilott. 

3 26 79 33 4 ^C^9 



of 25c man-hours to develop a course-hour and a decrease in 
course duration of 30 percent) the course development man-hours 
are offset by a corresponding reduction in instructor man-hours 
when the course has been taught to approximately 3, IOC students. 
By contrast, when the cost avoidance is based on the combined 
decrease in instructor and student costs (assuming the pay and 
allowance rate of instructors is twice tha.t of students), the 
breakeven point Is approximately 850 students. Direct support 
personnel are also identified with program management, the 
operation and maintenance of laboratory equipments and media 
devices, and other instructional support functions, but no infor- 
mation is available concerning how these personnel requirements 

are affected by the introduction of computer-based instruction. 
h. 

The fourth study that treated student time as a cost of 
training (Polcyn, Baudhuin, et al. , 1977) investigated the use 
of CMI for training at duty stations instead of at training 
centers. Schools provide training at both initial and advanced 
skill levels. Both require transfer of students to schools that 
result in expenditures that are a cost of the training. Initial 
and advanced training, occurring between duty station assignments, 
involve an additional permanent change of station (PCS) transfer 
resulting in costs for one-way travel and movement of household 
goods. Advanced training that occurs during a duty station 
assignment involves a temporary duty (TDY) transfer and incurs 
costs for round-trip travel and per diem payments for the dura- 
tion of training. 

Data were presented that permit estimation of transfer 
costs (including per diem) per course: $^25 for advanced train- 
ing based on PCS transfer, $^00 for advanced training based on 
TDY transfer, and $1^0 for initial training based on PCS trans- 
fer. A significantly higher cost ($825) for PCS transfer for 
advanced training is oited by the Air Force in "USAF Cost and 
Planning Factors" (Air Force Regulation 173-10). To the extent 
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that CMI and CAI permit training to be provided at duty stations, 
the resulting decrease In relocation costs Is a true cost avoid- 
ance . 

E.6 PROGRAM DELIVERY: LABORATORY EQUIPMENT 

No data were found on these costs of Instruction. 

'E.7 PROGRAM DELIVERY: MEDIA DEVICES 

The cost of using media devices presents methodological 
problems that were not addressed In the literature reviewed. 
Media devices are long-lived. Independent of sub,1ect matter, and 
generally portable; these attributes Introduce some problems In 
determining the costs of their use In a course or course segment. 

Unit procurement cost data are easy to compile from such 
sources as published catalogues. Table E-6 contains unit cost 
and other data for broad groupings of equipments. The wide 
range of costs Is a function of device size and other features. 
Note that the range of costs shown encompasses an order of mag- 
nitude yet does not include devices sized for presentation to 
large groups, such as in auditoriums. Determining their costs 
of use, in general, requires further data regarding operating 
costs, failure rates, and repair costs. A second citation to 
failure rates is shown in Table E-7, but we could find no infor- 
mation on repair costs. Their full cost of use would also require 
data regarding lifespans in order to amortize initial costs, and 
only Hess and Kantar (1977) contained any information. 

The methodological problems arise in determining cost of 
use in a particular training situation. The first problem in- 
volves selection of devices. A variety of media are available 
for presentation of instruction, and different media can be se- 
lected for presenting the same subject matter in small units of 
instruction, e.g., individual lessons. As a result, a course 
(or segment of a course) may utilize an extensive mixture of 
media. With the variation in the cost of media devices noted. 
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TABLE E-6. MEDIA DEVICES: UNIT COST RANGES 
AND OTHER INFORMATION^ 



Media Device 


Range of 
Initial Cost 
(Dollars) 


1 1 f p 
Span 

f V p a r Q \ 
\ 1 c a r b J 


MTBF^ 
V nou r s ; 


Sound Movie Projectors 


$175 - 


1000 


6 


90 - 110 


Videotape Recorders/Players 


$600 - 


8000 


5 




Sound Slide/Str'o Projectors 


$100 - 


1000 






Silent Movie Projjcitors 


$150 - 


250 






Silent Slide/Strip 
Pro jec torsb 


$ 25 - 


900 


6 - 10 


90 - 150 


Random Access Slide Projectors 


$500 - 


2000 






Ml crof 1 Im/Fiche Readers 


$ 80 - 


800 






Aud i otape/Di sc Players 


$ 30 - 


325 






Teaching Machines (Individual) 

Audio Visual 
Rate Control 
Constant Control 


$230 - 1000 
$1950 






Visual 

Rate Control 
Constant Control 


$140 - 
$220 - 


380 
1 200 






Audio 

Rate Control 


$190 - 


470 







^Excludes equipments too large for use in individual classrooms. Costs 
are for commercial quality equipments. 

'^Includes overhead projectors. 
Mean time between failures. 

Source: Hess and Kantar, 1977. 
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TABLE E-7. MEDIA DEVICE RELIABILITY 



Med1 a Device 


Hours of 
Use 


Number of 
Fai 1 ures 


MTBF^ 
(Hours ) 


Mot i on Pi cture Projector 


2,364 


12 


197 


Videotape Player 


2,315 


5 


463 


Til ft « • , 

TV Monitor 


2,315 




2,315 


Sound/Fi Imstri p Projector 


1 02 ,509 


] 25 


820 


SI i de Projector 


3,711 




3,71 1 


oiient riimstnp Projector 


1 2 , 427 


33 


377 


Microfiche Reader 


30,636 


11 


2,7C5 


Audio Tape Player 


33,394 


12 


2,783 


Headset 


136,199 


5 


27,240 



Mean time between failures. 

Source: AIS Integrated System Test (Draft). McDonnell 
Douglas, 1977a. 

different mixes of media can Imply sizeable differences In course 
costs. Determination of the device mix Implies an extensive 
course design effort. Including specification of equlpment-to- 
student ratios and the environments In which different equip- 
ments will be used (e.g. ^ Individual or auditorium presentation). 
The systematic Investigation of alternative course designs to 
determine cost differences associated with these mixes could be 
£m expensive process. 

A second problem arises from the physical nature of media 
devices. Even If a mix of media devices were formulated for a 
training course or course segment, the cost of using those de- 
vices In that course must still be defined. Media devices are 
long-lived and Independent of subject matter; once procured, 
they serve as an Inventory to satisfy requirements levied by all 
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courses offered at an Installation. Operations and maintenance 
costs may be based on Individual course usage In a straight- 
forward manner. However, the full cost of the use of media de- 
vices will depend upon whether or not existing stocks are suffi- 
cient to meet the demands of all users. If stock levels are 
adequate, equipments will be available for proposed courses (the 
Incremental user) with no additional outlays; if not, procurement 
of additional equipments would be Indicated and these purchase 
costs niust be accounted for in some manner. Information regard- 
ing the ajiequacy of existing inventories and user requirements 
is rarely available, and there are a number of ways in which 
purchase costs can be apportioned. Analyses must resort to 
assumptions and allocation schemes that are essentially arbitrary 
and the results would be dependent upon Just what assumptions and 
schemes were employed. 

E.8 INSTRUCTIONAL PROGRAM DELIVERY: COMPUTER SYSTEMS 

Information on instructional computer hardware costs is un- 
expectedly hard to find in the literature, despite the central 
role it plays in system capability and costs. Computer hardware 
appears to account for no more than a modest share of computer- 
based instruction system life-cycle costs. However, a purchase 
of hardware represents the bulk of early system cost and amounts 
to a commitment to CBI and the other costs and risks that commit- 
ment entails. 

Substantive information was found on five hardware systems 

PLATO IV, TICCIT, GETS, IBM 1500, and Navy CMI; some Information 
is available on a one-of-a-kind configuration assembled to eval- 
uate shipboard use of CMI. One additional source contained a 
single aggregate figure for development of the Air Force Advanced 
Instructional System (AIS) . However, this figure represented a 
contract value for a mixture of hardware, software, and course- 
ware development that could not be separated by function. 
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Taken system by system, the detail and coitn ^ ' of the 

data are diverse. In no case was the material &a. ' for 
an adequate understanding of the drivers of system c Indi- 
vidual studies are typically limited in scope, and t\.-> : >st 
information presented is generally limited to only that necessary 
to the principal issue addressed. For example, in ex; e/i*imental 
programs, the costs that were reported were normally :i.imited to 
those directly (and incrementally) incurred as a resulT of the 
program, i.e., those that could be identified as budget expendi- 
tures of the demonstration. 

The discussion that follows is organized according to com- 
puter system rather than type or location of hardware. Note 
that all costs are displayed as then-year dollars rather than 
being adjusted to a common base period. In the light of chang- 
ing CAI/CMI hardware conf iguratios and technology, no satisfac- 
tory price index could be found. In several studies, it appeared 
that costs given were not current with the publication date, and 
no information was provided as to the applicable dates of the 
costs . 

E.8.1 PLATO IV . 

More studies have addressed PLATO IV than any other system. 
Altliough purchase of a PLATO IV system is very expensive, access 
to its services can be purchased in 5mall and divisible units, 
e.g.y the individual terminal, with a small initial outlay, and 
its software system«^is highly developed. These features are 
particularly attractive for small-scale programs, typical of 
experimental applications. 

Most military use of PLATO IV was supported by a contract 
between DARPA and the military Services with the Computer-Based 
Education Research Laboratory (CERL) at the University of 
Illinois. This arrangement, however, casts some question on the 
value of cost data reported. In addition to being non-profit, 
CERL is the originator of PLATO IV and has a vested interest 
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in its use. The CEHL contract charges are cqn^derably le^s 
than those of Contrcl Data Corporation (CDC), currently provid- 
ing PLATO IV services on a commercial basis; the CERL chaj^ges 
may well be less than the military could provide the same 
services for on an m-house basis. The values shown in Taple 
E-e are reported actuals from experimental programs funded 
through the CERL contract for Hurlock and Slough, I976; U.^. 
Army Ordnance Center and School, 1975, Dallman, DeLeo et at., 
1911 \ and Steinkerchner , Delgnan, et al., 1911. The valued 
shown for Kribs, 1976, for terminals and PLATO IV system support 
and the system suppoi't shown for Crawford, Hurlock, et al. , 1976 
are quotations from CDC. 

The costs displayed in Table E-8 are based on the use of a 
small fraction of the central hardware capability of a FLAfO IV 
system. The computer access is charged to cover central system 
operating costs, amortization of the hardware, and other services 
provided by CERL on the basis of the number of terminals con- 
nected. Large-scale use of PLATO IV by the Services would j^rob- 
ably involve purchase of both terminal and central processor- 
hardware with a sizeable initial outlay. 

Control Data Corporation has provided an estimate of hard- 
ware procurement costs for a 1000-terminal system capable or 
supporting 725 terminals simultaneously in a student mode, or 
^25 in an author mode. The configuration and component costs 
are shown in Table The simultaneous terminal constraint Is 

a function of central processor cycle time rather than the "swap" 
time between the random access and swap memories. Saturation of 
a swap memory capacity is reached at approximately 800 students; 
this level restricts student blocks to iJOOO words when all stu- 
dents are accessing different programs. 

As shown in Tabje E-9, the configuration contains components 
not currently available on a commercial basis. The extended swap 
memory (ESM) , current^ly in development, is of metal oxide semi- 
conductor (MOS) technology and will employ a controller witn 
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TABLE E-8. COSTS ASSOCIATED WITH THE USE OF PLATO IV 



Ittfll 


Huriockand 
Slough. 1976 


Crawford, 
Hurtock. 
et al., 1976 


U.S. Army Ordnance 
Center and School. 
1975 


Daflman. DeLeo, 
etal, 1977 


Steinkerchner. 
Deignan, 
1977 


Kribs, 1976 


f^umbcr of TBfmindis 
On Line 


12 


16 


14 


X 


20 


12 


See Note 


1.2 


2 


1 


1.3 


1 


2.4 




$ 


Code 


$ 


Code 


$ 


Code 


$ 


Code 


$ 


Code 


$ 


Code 


PUTD IV Terminal 
Basic 

Touch Panel 
Slide Option 
Audio Unit 


10,500 


PJ 
1 
! 


10,500 


PJ 
I 






5.205 
544 

2.000 


PJ 
PJ 

PJ 


5,000 
410 
600 

2,000 


PJ 
PJ 
PJ 
PJ 


11.650 


X.P,T 

1 

1 


PUTD W Sfstem Support 

Computer Access 

Maintenance) and 
Other 


2.500 
1.300 


T.CJ 
T.C.Y 


35,820 


Q.4T 


2.500 
1.000 


TX.Y 
T.C.Y 


1.500 
l.OQO 


T.C.Y 
T.C.Y 


1.500 
1,000 


T.C,Y 
T.C.Y 


?.2O0 
1.380 


C,X,T,Y 
C,X,T,Y 


Communications 
Commercial 
Lease Line 

GSA Lease Line 
Microwave System 
Multiplexor 
Data Port 


.50 
10.500 


M.4T 
P.4T 






.50 


M.4T 


204 
9,630 


T 
P 


.61 

4.680 
300 


M.4T 

P.4T 
P.4T 


.95 
6,300 


M,4T 
C,X,4T,Y 


Virian Printer 














7.586 


P 











Code: C. Contract service or lease. M: Per milel month. 

P: Purchase. T: Per terminal. 

X: Quotation 4T: Per four terminals. 

I: Included in "basic" cost. Y: Per year. 



Notts: 

1. These studies report the results of small scale evaluations of the PLATO IV system that were partially funded by OARPA. Since the programs 
were complete at the time of documentation all costs shown are assumed ir) be actuals except thnse noted as estimates. OARPA support 
included all items shown in the table, and one would expect to find the crjsts of like items trj he the same. The higher maintenance cost nntcd 
in Hurlock and Slough. 1976. is explained by the longer distance between San Oiegn and thf: University of Illinois, but the documentation prrjvidcd 
no explanation for the other diflerences. 

2. The experimental program addressed in Crawford, Hurlock. et a!.. 19?6. is one of a number of programs addressed m Hurlock and Slough, 1976. 
The lower PLATO IV system support costs shown by Hurlock and Slough. 1976. is based on the CERL contract while the higher value shown in 
Crawford. Hurlock. at aL 1976, is a quotation soliated from COC. The estimates shown in Kobs, 1976. for both the terminal and system support, 
were obtained from CDC. 

3. Communications for this program were achieved by microwave for transmission from CERL to Chanule AFB la distance o! 15 miles) and local 
telephone lines for transmission from Chanute to CERL 

4. The basic terminal is noted as including a local MOOEM. 
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TABLE E-9. PLATO IV HARDWARE PROCUREMENT 
AND MAINTENANCE COSTS 



Central Site Equipment 

Central Processor 

Extended Swap Memory 

Computer Interface Unit 

Computer Interface Unit Expansion 

Tape Controller 

Tape Drive 

Disk Controller 

Disk Drive 

Line Printer 

Site Controller 

Remote Interface Module 

Mu) tiplexDr/MOOEM 
Total 



Model 
Number 



4001 
4001-2 
721-21 
667-2 
7054-41 
844-41 
580-12 
40002-1 
40002-13 



Year of 
Avai la- 
bility 



1978 
1981 
1978 
1978 
1978 
1978 
1978 
1978 
1978 
1978 
1978 
1979 



Quanti ty 



20 
1 

22 
176 
125 



Purchase 
Price 
(000) 



S1529 
1351 
14 
12 
61 
41 
297 
515 
71 
213 
211 
688 

$5003 



Monthly 
Maintenance 
Cost 
(000) 



Remote site Equipment 
Terminal 

Multlplexor/MOOEM 
Total 



IST2 



1980 
1979 



1000 
125 



$5000 

68a 

J5 68_8 

Source: Unpublished matcMals received from and conversations with R A Moe 
Notes: 

1. CYBER 174-6 with 131K words of random access memory. 

2. MOS technology with 4'million word capacity. 

3. Capable of controlling 256 active terminals. 

4. Each expansion unit is capable of controlling 256 active 
controlled by the master computer interface unit. 

5. Capable of controlling 3? active terminals. 

6. Capable of controlling 4 active terminals {i.e., eight per site control 

7. Eight-port multiple^^or and 9600 bps MOOEM. 

a. P^ojuction version of the current terminal. Principal changes to permi 
^ib^win Vs:ane?! ''''''' '''''' '''''' ^lytolVllTe 



$ 4.6 
2.0 
0.1 
0.0 
0.3 
0.3 
1.0 
2.1 
0.3 
2.8 
0.9 
3.1 

J17.5 



$2 5.0 
3.1 

$28.1 



of Control Data Corporation. 



terminals after the first 256 temiinals 



ler). 



t production of larger lot 
integral to the case, and 



E-23 

200 



eight tiniv'js the addressing capability of those used with the 
current magnetic core units. The higher capability controller 
will permit ESM growth to 16 million words, compared with the 
2-million word limit of the current controller. The terminal is 
a smaller integrated unit with emphasis placed on producibility 
of larger lot sizes. The eight-port multiplexor is currently in 
pilot use/ Its impact on system costs lies in the potential of 
reducing the number of multiplexors and MODEMS and the costs of 
communications services by up to one-half. With one exception, 
the priced shown are current values rather than projected values. 
The one e^cception is the extended swap memory where the recent 
trend dec:Line in MOS chip cost?; has been pro,1ected to reflect 
anticipated I98I costs. 

E.8.2 TICCIT. 

Very little information was found concerning TICCIT hard- 
ware cost^ in the literature. Much of what was found is out of 
date as tpe result of a recent major configuration change insti- 
tuted by liazeltine Corporation, which currently markets TICCIT 
on a commercial basis. Among other changes, the new configura- 
tion substitutes the Nova 3/D central processor for the Nova 
800 processor. 

One description of TICCIT hardware and costs is shown in 
Table E-10, (Kearsley, 1977 )• This information was obtained 
from the MITRE Corporation in 197^ and pertains to the Nova 800 
configuration. Kribs (1976) provides an estimate of $270,000 
for hardware and $130,000 for system installation and checkout 
for a 13-terminal configuration. This figure is in close agree- 
ment with the MITRE information and is based on the NOVA 800 
configuration installed at North Island Naval Air Station. 

Unpublished information received from Hazeltine regarding 
the new oonf iguration indica\.es that the central system has a 
degree of modularity and may consist of one or two central pro- 
cessors, ciepending upon the number of terminals to be supported. 



TABLE E-10. ESTIMATED COST OF TICCIT HARDWARE 
(The Mitre Corporation, 1974) 



Central Processor Unit 






Main Processor 




$34,000 


Terminal Processor 




15,000 


Disc Drive & Controller (3 at 


$23,300 each) 


70,000 


Tape Unit 




9,000 


Card Reader 




4,000 


Line Printer 




11 ,000 


CRT Terminal 




3,000 


Computer-Computer Line 




3,000 


Character Generator 




7,000 


Keyboard Interface 




6,000 


Audio System 




56,000 


Refresher Control 




6,000 


Crossbar Switch (Audio/Video Switching) 


17,000 


Video Tape Players (20 at 850 


each ) 


17,000 


Cabinets 




7,500 




Total 


$265,500 


Terminals 






TV Monitor (including 


128 at $320 




modification) 




Keyboards 


128 at $170 




Refreshers 


128 at $735 




Signal Processors 


128 at $125 




Total 


$1350 


$172,800 


Total 




$438,300 



Source: Kearsley, 1977 
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and varying numbers of disc units , depending upon the extent of 
courseware and number of students to be supported. Hazeltine has 
provided what is described as "budgetary pricing information" as 
of June 1977, as shown in Table E-11. Hazeltine indicates that 
as of June 1978 these prices had increased approximately 20 per- 
cent, bringing the hardware cost of a 128-terminal system to 
well over $1 million. 

E.8.3 IBM 1500 

Three references to IBM 1500 system costs were found in the 
literature, and one included the IBM 1460 system, as summarized 
in Table E-12. Two sources provided tabular breakouts (Kopstein 
and Seidel, I969 and Kearsley, 1977) while the third (Ford, 
Slough, and Hurlock, 1972) provided a single "bottom line" cost. 
Only Kearsley (1977) explicitly stated that the costs came from 
accounting records of operating experience. 

Two points are to be noted regarding this material. The 
first is that the hardware costs are monthly lease values in all 
cases, with maintenance provided by t^e contractor. The second 
is that the lease costs are roughly the same, although the publi- 
cation dates span a 10-year period of rising general cost levels 
and falling costs of computer service. The discussions in the 
references were insufficient to explain the similarity, but some 
rationale may be gleaned from the different environments on 
which the estimates are based. The values provided by Kearsley 
(1977) are based on an extended period of operation that began 
in the late 1960s; it was not noted to which part of the time 
period the values pertain. Further, the system was installed at 
a Canadian university where relative costs may differ from those 
in the United States. Data provided by Kopstein and Seidel (I969) 
are based on civilian installations, while those of Ford, Slough, 
and Hurlock (1972) are based on military use. With the different 
ways data categories are combined, it is possible that central 
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TABLE E-11. ESTIMATED COST OF TICCIT SYSTEM 
(Hazeltine Corporation, 1977) 



Basic Systems 






System I (32 Terminals) fanaritv 




^ D , UUU 


System II (64 Terminals) Capacity 




0 O 0 , u u u 


System III (128 Terminals) Caoacitv 




O 0 U ) u u u 


Options 







Graph i c Digitizer 




25,000 


Disc Drive 




21 ,000 


Software Support Package 

(Not available for System I) 




2d nnn 


Optical, Spares Package 




21 nnn 


Video Tape Option - System I 




AC nnn* 

HO J UUU 


S V ^ t pm T T 




oO , 000* 


System III 




90,000* 


Audio Option - System I 




100,000 


System II 




100,000 


System III 




70,000 


Terminal: Average Cost for Quantity 


: 32 


2,375**' 




64 


2,325** 




128 


2,275** 


^Deduct $20,000 when audio option is 


also 


i ncl uded . 



*Add $250 for 1 ight pen. 

Source: Hazeltine Corporation, 1978. 
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TABLE E-12. IBM 1500 SYSTEM COSTS 



Item 


Source 


Kopstein 
and 

Seidel , 
1969 


Ford , 
Slough , 
and 

Hurl ock , 
1972 


Kearsley , 
1977 


Date of Installation 
System Designation 
Number of Terminals 


1460 
32 


1500 
32 


1500 
32 


1968 
1500 
32 


Monthly Hardware Lease Cost 
Central Processor 
Data Storage 
Termi nal s 
Mai ntenance 
Total 


$ 5, goo 

2,800 
j 4,600 


1 $14,800 




$12,000 


j $ 6,700 

4,648 
1 ,884 


$13,300 


$14,800^ 


$12,000* 


$13,242 


Monthly Operating Cost 

Number of Personnel 

Operators' Salaries 

Programmers'/Managers' 
Salaries 

Total 


1 

700 


1 

700 






6 

$ 3,198 
3,968 


700 


700 




$ 7,166 



Includes Maintenance. 



^ J 
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processor lease rates might have declined and maintenance costs 
Increased over time and without this being noticeable. 

E.8.4 Navy CMI System . 

The Navy CMI system has used two different central hardware 
systems. Initial development and application of the system was 
accomplished using a Xerox Sigma 9 system, located at Memphis 
State University and dedicated to Navy use during regular train- 
ing hours. Later, the Navy procured a Honeywell Series 60 Level 
66 system. The configuration, shown in Table R-13, is described 
in Hansen, Ross, et al . y 1975. At the time of publication the 
hardware was under contract for procurement, and we have not 
verified whether configuration changes occurred between then and 
its final installation. Costs of its components were obtained 
from issues of computer price survey publications, as noted. 

Student terminal configuration and cost information is 
taken from Polcyn, Baudhuin, et al. , (1977) and is shown in 
Table E-lH . The date of this publication would allow these 
values to be based on historical records that antedate installa- 
tion of the Honeywell central processor. With regard to main- 
tenance costs, neither the philosophy (in-house or contract) nor 
a breakdown between labor and material was given. 

E.8.5 General Electric Training System (GETS) . 

GETS (also known as Computer-Based Training System or CBTS) 
is a stand-alone (single terminal) unit. The only military appli- 
cation at present is for crew training for the TRIDENT system 
v/hen there are more students than the regular laboratory equip- 
ment can handle. A life-cycle cost evaluation (Kribs, 1976) 
estimated unit hardware production cost to be $3^1,000 and other 
nonrecurring costs associated with procurement of 13 units 
(spares, manuals, installation, etc.) at approximately $125,000. 
These values are based on a contractor quote that is several 
years old, and its timeliness is open to serious question. 

E-29 

2')6 



TABLE E-13. NAVY CMI SYSTEM: INITIAL CENTRAL SITE HARDWARE COST 



Component 


Model 
Number 


X u a II L 1 u jr 


Unit Cost 
(000) 


^ n II r p 


Total Cosi 
(000) ' 


Co nt rA ^ PrnroQcnr fl^11^ '^^-hit d*h\/tp wnrHc:^ 


CPS 


6202 


2 


$743 


0 


2 


$1486.0 


PomnujiKlo Hicp lln^tc 
i\ClliUVaUlC UI^C Ullll*^ 


MSU 


0400 


8 


21 


9 




175.2 


Unit Record Processor (disc) 


MSP 


0600 


2 


6^ 


9 


1 


123.8 


Magnetic Tape Processor 


MTP 


0600 


1 


25 


.7 




25.7 


9-Track Tape Drive (800/1600 cpi ) 


MTU 
MTF 


500/ 
0012 


4 


20 


.6 




82.2 


7-Track Tape Drive (556/800 cpi) 


MTU 
MTF 


410/ 
0116 




15 


.5 




15.5 


Unit Record Processor (peripherals) 


URP 


0601 




19 


.6 




19.6 


Line Printer (1200 1pm) 


PRU 


1200 




44 


.4 




44,4 


Card Reader (1050 cpm) 


CRU 


1050 




19 


.2 




19,2 


Card Punch (300 cpm) 


CPZ 


0201 




34 


.0 




34.0 


Console 


CSU 


6001 


2 


36 


.2 


3 


36.2 


Datanet-Front End Processor 


DCP 


6632 


2 


124 


.9 


2 


249.4 


System Total 










2311 .2 



Unit Cost Sources: 1. Computer Review 1978 , Vol. 18, No. 2. 6ML Corp., 

Lexington, Mass. 1978. 

2. Datapro 70 . Datapro Research Corp. (Subsidiary 
of McGraw Hill ) . Delron, N.J., 1 977 . 

3. Authorized ADP Schedule Price List . Federal System 
Operations, Honeywell Information Systems. McLean, VA., 
Undated. (6SA price list, FY 1978). 

Configuration Source: Hansen, Ross, et al., 1975. 
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TABLE E-14. NAVY CMI STUDENT TERMINAL COST 



Item 


Un i t 
Initial 
Cost 


Unit 

U II 1 L 

Annual 
rid intenance 


Opscan 17, Optical Scanner 


$ 8,998 


$ 82.8 


Opscan Auto-Feed 


6 5? 


60 


GDC-202-9D Modem 


400 


60 


Terminet 1200, Keyboard/Printer 


4,200 


816 


Total 


$14,250 


$1 ,764 



Source: Polcyn, Baudhuin, et al., 1977. 



Technological advance in the intervening period has had its 
greatest impact on the types of components used in stand-alone 
systems; in particular the costs of microprocessors and both 
integral and peripheral data storage devices have experienced 
relative declines. 

E.8.6 Shipboard Computer-Based Instruction . 

Shipboard use of automated data processing for non-tactical 
applications has been investigated by the Navy for about 10 
years. Two areas of application have been investigated under 
the designation "Automated Shipboard Information Management 
System (ASIMS)". The first is a general data management capa- 
bility "Command Management System (CMS)", and the second is 
computer-based instruction, "Computer Integrated Instruction 
(CII)". The first shipboard test, a Data General NOVA 1200 
system installed aboard the USS Dahlgren in 1973, emphasized 
Command Management System applications. In 1975, the system was 
transferred to the USS Gridley where emphasis shifted to Com- 
puter Integrated Instruction applications. The NOVA 1200 has 
now been replaced with a Digital Equipment Corporation (DEC) 
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PDP 11/60, and the application has been extended to include both 
CMS and CII. 

Both the NOVA 1200 and PDP 11/60 are considered mini-com- 
puters. Little information was found concerning the configura- 
tion or cost of this installation. The configuration displayed 
in Table E-15 is containec3 in unpublished information received 
from the Navy Personnel Research and Development Center (undated) 
and pertains to the USS Gridley. Component costs were contained 
in a discussion of the USS Dahlgren installation in Middleton, 
Papetti, and Micheli (197^). There is an ambiguity between 
these two sources concerning whether the costs shown for the 
disc drives and CRTs represent unit costs or the costs of two 
and four units, respectively. In Table E-15, they are treated 
as costs of the quantities shown; if they should have been 
treated as unit costs the total installation cost would rise to 
over $100,000. Note that the cost values are based on 1973 in- 
formation, and it is questionable whether they represent the 
current costs of mini-computers of like capability. 

TABLE E-15. HYPOTHETICAL SHIPBOARD SYSTEM: ESTIMATED 



HARDWARE PROCUREMENT COST 


Component 


Cost 


Central Processor (NOVA 1200, 32K Memory) 


$20,500 


Line Printer and Controller (365 1pm) 


13 ,100 


Card Reader and Controller (100 cpm) 


6,050 


Disc Drive and Controller 

(2 units, 12 million 16-bit words each) 


24,000 


Alphanumeric CRT (4 units, Hazeltine 2000) 


3,000 


Other (Cassette Tape Drives, Teletype) 


2,000 


Total 


$68,650 



Source: Unpublished information received from the Naval Personnel 
Research and Development Center (undated); also 
Middleton, Papetti, and Micheli, 1974. 
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E.8.7 System Cost Comparison . 

The hardware systems discussed In this Appendix Include the 
extremes of current system capabilities In terms of the number 
of terminals supported. The cost of these systems can be ex- 
pressed In three ways: (1) system procurement cost, (2) system 
procurement cost per terminal,' and (3) system procurement cost 
per student-hour based on assumed life spans and utilization 
rates. Table E-16 displays the comparisons between each system 
for each of the three measures. The inverse relationship between 
system size (or Initial procurement cost) and cost per connected 
terminal or per student hour Is rather consistent within the data 
sample, but we have strong reservations regarding the reliability 
and timeliness of the data and of drawing strong conclusions 
based on them. Although It appears safe to say that computer 
hardware for the Navy CMI Instructional system costs less per 
student hour than the computer hardware for any of the Cfil In- 
structional systems. It should also be obvious that other computer 
system costs, such as for Installation and maintenance, are not 
Included In these values. 

E.9 PROGRAM DELIVERY: COMMUNICATIONS 

Communications costs arise only In large computer-based 
systems where terminals may be located at appreciable distances 
from central processors. Currently, this applies only to PLATO 
IV and the Navy CMI systems. Four methods of communication have 
been suggested— land lines, microwave systems, satellites, and 
closed-circuit television. All applications of these systems, 
with the exception of one PLATO IV experimental program, have 
employed land lines. The program pursued at Chanute AFB, re- 
ported in Dallman, DeLeo, et at., 1977, utilized a microwave 
system as a primary communication, link. 

The PLATO IV programs that utilized land lines employed GSA 
leased lines with costs of roughly $.50 per mile per month. This 
rate was also cited in several other studies where it was also 
reported to be approximately one-half of the commercial Interstate 
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TABLE E-16. COSTS OF COMPUTER SYSTEMS HARDWARE 



Method 
of 

Instruction 


Computer System 


Central 
Processor 

Cost 
(Thousands) 


Terminat, 
Unit Cost 
(Thousands) 


System 
Hardware 

Cost 
(Thousands) 


System 
Hardware Cost 
Per Terminal 
(Thousands) 


System 
Hardware Cost 
Per Student- 
Hour^ 




IBM 1500 
32 Terminals^ 






$ 800 


S 25 


S 2.49 




PLATO IV 
1,000 Terminals^ 


S 5,000 


S5.7 


10,700 


11 


1.48*^ 


CAI 


TICCIT 

32 Terminals' 


760 


2.9 


850 


27 


2.66 




64 Terminals 


870 


2.8 


1,050 


16 


1.64 




128 Terminals 


970 


2.8 


1,330 


10 


1.04 




GETS 
One Terminal 






34 


34 


3.40 


CMI 


Navy CMI 

6,000 Students' 

16.000 Students^ 


2,300 
2,300 


14.3 
14.3 


4,020 
6,880 


34 
22 


0.07 
0.04 



4-18-79 



'2.000 houn per terminal per year lor S years. 

^ncludei maintenance. Baieil on Icaie ratei and amortizing equipment over a S-year period, 19^7, 1972, 1977. 

^Control Data Corporation quotation, from private communication dated 14 August 1978. 

'^Bjied on 725 active terminal conitralnt. 

'Hazeltine quotation, from private communication, 1978. 

^20 terminali at SO itudenti per terminat. 1977, 

9320 terminats at SO studenti per terminal. 1977. 

3 26 79'3S 
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rates. In fact, the commercial tariffs are based on complex rate 
structures set by the Federal Communications Commission for inter- 
state (or long) line$ and individual states for wholly Intrastate 
(or short) lines. 

The rate schedule (Table E-17) for commercial long lin^s v/as 
found in Ball and Jamison, (1973) and Middleton, Papetti, and 
Micheli (197^): 

TABLE E-17, RATE SCHEDULE FOR COMMUNICATIONS 
OVER COMMERCIAL LONG LINES 



DISTANCE 
INTERVAL 
(MILES) 


COST PER INCREMENTAL 
MILE PER MONTH 


AVERAGE COST PER MILE 
PER MONTH (AT LIMIT OF 
DISTANCE INTERVAL 


1-25 


$3.30 


$3.30 


26-100 


2.31 


2.56 


101-250 


1.65 


2.01 


251-500 


1.15 


1.58 


>500 


0.83 ' 


1.20a 



>At 1000 miles. 

Source: Ball and Jamison (1973). and Middleton. Papetti. and MicheU (1974). 



In addition to the line rates, "conditioning" and "terminat^.on" 
charges of approximately $90 per line per month are levied. On 
the basis of this schedule, it can be seen that close to trans- 
continental distances of greater than 2,000 miles are required 
before the average rate approaches one dollar per mile. On:J.y 
one paper (Ball and Jamison, 1973) addressed the cost of shtprt- 
line communications. These rates were characterized as a ctpn- 
stant function of mileage that approximates $^ per mile per 
month plus line conditioning and termination charges of app^*oxi- 
mately $125 per month. 

Ball and Jamison, (1973) was also the only paper that 
addressed the costs of equipment associated with land-line 
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communications. Since the document was not concerned with par- 
ticular hardware systems, it provided only typical costs for 
generic types of equipments with no references to sources or 
pai*ticular equipments on which the estimates v/ere based. The 
limits of these estimates are shown In Table E-18. 



TABLE E-18. COSTS OF LAND-LINE 
COMMUNICATIONS EQUIPMENT 



Equ i pment 


Unit Cost 


(Do! lars ) 


Mul t i pi exors 




8 Channel 


$ 1,600 


Centra 1 


7,000 


System 


10,000 


Modems 




1200 Baud 


500 


,2400 Baud 


1 ,750 


4800 Baud 


5,400 


Multiplex Computer 


9,000 



EJO PROGRAM DELIVERY: MATERIALS 

No data were found for these costs of instruction. 

EJI PROGRAM DELIVERY: FACILITIES 

Furnished instructional facilities are a requirement of any 
formal training program and, given a static-sized force, it may 
b^ considered that existing facilities would generally be avail- 
able without further expenditure for commonly used assets. How- 
ever, the introduction of self-pacing imposes a requirement for 
assets not associated with traditional instruction, and the 
introduction of PLATO IV (or any other computer-based system) 
imposes other requirements. These are properly costs of the 
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newly employed training technology, i.e., CAI or CMI. Two USAF 
PLATO IV demonstration programs— Dallman, DeLeo, et at., I977, 
and Steinkerchner, Deignan, Waters, and DeLeo, 1977, reported* 
costs of facility modifications and furnishings required for 
instruction. 



" 



In the cag^e of these two USAF demonstrations, only two 
facility items fell into this class. The first is individual 
learning carrels and the second is the provision of electric and 
pneumatic outlets at the carrels to service the PLATO IV termi- 
nals. Other requirements such as communication links would 
appear to be required al&o, but these were not listed. The cost 
of carrels was reported at $90 per unit in one study and at $260 
per unit in the other. No descriptions were provided to explain 
such a wide difference, but it is of note that Hess and Kantar, 
(1977) reported an average or typical cost of carrels for pro- 
grammed instruction at $160. 

The cost of installing electric and pneumatic service can 
be expected to vary widely as a function of the characteristics 
of the building in which they are installed. In the case of 
these two programs, service for 30 carrels was installed at an 
average cost of $6l per carrel and service for 20 carrels was 
installed at an average of per carrel. In neither case was 

a breakdown between electric and pneumatic provided, and in 
neither case was it noted whether pneumatic service was already 
available in the building or whether the program involved pro- 
curement and installation of a compressor as well as the running 
of the compressed air lines. 

E.12 PROGRAM MANAGEMENT AND ADMINISTRATION 

No data were found on these costs of instruction. 
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E.13 STUDENT PERSONNEL 

Dat^L on costs of Student Personnel are Included in Section 
E.5, Program Delivery: Instruction, Including Instructors and - 
Instruction Support Personnel and Student Personnel. 
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